Context

A user connects to a remote system with SSH via some terminal. But after connecting, the terminal ouput is filled with messages and more importantly the BASH session is inoperative.

Thus we must troubleshoot this issue, seemingly linked to a problem with the BASH invocation - potentially including multiple files.

Disclaimer

Even though we focus here on "GNU Bourne-Again SHell" (referred to in this document as BASH or bash), following elements may be useful for other Unix shells. The use of strace would most certainly be helpful in even more troubleshooting situations.

Also note that the BASH part itself may differ depending on the involved (Unix) system(s). Consequently, one must refer to the system local manual(s) beforehand (aka "Read The Fine Manual" - RTFM). And that is indeed the very first thing we do in the previously introduced context.

BASH Invocation

Let us start with the BASH manual - more precisely the page section ¹ INVOCATION of bash main section (man.1).

In this manual page section, one may read about:

• login and interactive shells.
• startup files execution.

We summarize these explanations in the subsections below. But do keep in mind that situations may be more intricate - in that event, we shall firstly refer back to this manual.

Login and Interactive

To sum it up, BASH can be invoked as a shell with 2 different and independent "modes":

• login (or not - referred afterward as "non-login"): depends if a user has been identified for this shell (login) or not (non-login).
• interactive (or not - referred afterward as "non-interactive"): mainly depends if the shell is reading input from a terminal / user (interactive) or from a file (non-interactive).

Theses modes may be enabled with respective options for bash:

• -l (or --login) for login mode.
• -i (no long format) for interactive mode.

Thus, regarding these 2 modes, there are 4 possibilities in total for a BASH shell:

• non-login + non-interactive (eg: a basic script).
• login + non-interactive (eg: a more advanced script requiring some specific startup files).
• non-login + interactive (eg: a nested shell - bash executed without option in a parent interactive login shell).
• login + interactive (eg: a user remote connection).

Regarding the provided examples, they must not be taken for granted - as we shall see afterward, conclusion may differ from our first assumption.

We therefore need a precise way to determine these modes for a BASH shell. But it is not directly (via a single command) possible to know if one (or both) of these modes are enabled.

However, for the interactive mode, the bash manual (section 1, subsection INVOCATION) can (once more) help us:

PS1 is set and $- includes i if bash is interactive, allowing a shell script or a startup file to test this state.

$- is the bash parameter containing the option flags specified during the BASH invocation. So, to check if a shell is interactive:

[[ $- == *i* ]] && echo 'Interactive shell' || echo 'Non-interactive shell'

For the login mode, the bash manual seems more vague. Notably, there is no specific value in $-.

Fortunately, the shopt command comes to our recue. This command can set or unset shell options, but also list the current shell options. Especially, the exit status ($?) informs if the shell option is enabled (0) or disabled (1).

And there is a shell option linked to the login mode: login_shell. Thus, to check if a shell is a login shell:

shopt -q login_shell && echo 'Login shell' || echo 'Non-login shell'

Note for shopt that:

• -q option simply suppresses the output (only exit status via $? is required).
• there is no shell option corresponding to the interactive mode (hence our previous other method).
• there is no manual, but some help is available with --help option.

Startup files

When BASH is invoked, it reads different files, in a specific order, to configure the shell profile.

Depending on the "mode" described above, the files and/or the order may vary.

In our use case, we are interesting by troubleshooting the problem met by an user connecting to a remote system with an interactive login shell.

The BASH manual describes the following startup files for this case:

• /etc/profile
• ~/.bash_profile or ~/.bash_login or ~/.profile (in that order, only the first one that exists and is readable).

Just for the sake of comparison, for an interactive non-login shell, the (completely different) list of startup files is:

• /etc/bash.bashrc
• ~/.bashrc

Finally, note also that some files are read when exiting a shell (eg: ~/.bash_logout for an interactive login shell).

"Naive" (faulty anyway) approach: manual lookup for sourcing

Without the knowledge from BASH manual, one may at first think about dealing with the case this way: one cannote that (or simply know) that the user files are executed during BASH invocation ; thus, one can look for all the loaded files.

1. "Naive" version (with bash -x) [fn:2]:

$ bash -lixc exit 2>&1 | sed -n 's/^+* \(source\|\.\) //p'

Or alternatively with awk rather than sed for example:

$ bash -lixc exit 2>&1 | awk 'match($0, /^+* (\.|source) (.+)/, s) {print s[2]}'

But with this "simple" version, we only look for sourcing done with specific methods (source, .). Some more exotic methods may be missed (some example?).

Furthermore, we miss the file automatically sourced by the system (i.e. not sourced from another sourcing file) - most notably the first sourced file (cf. comparison further).

"Cleverer" (at least more precise) approach: list opened files with strace

$ echo exit | strace bash -li |& grep '^open'

Or a maybe more complete alternative:

$ strace -t -e trace='openat' --decode-fds -e signal=none bash -lic exit 2>&1 | grep -P 'openat.+(?<!\(No such file or directory\))$'

Here, with strace, we are sure to list all files that the system uses when Bash is initializing. We have more output to filter, but this method is more precise.

Comparison:

For a login non-interactive Bash session.

$ bash -lxc exit 2>&1 | sed -n 's/^+* \(source\|\.\) //p'
/etc/profile.d/bash_completion.sh
/etc/profile.d/debian-mate-default-settings_gtk-accessibility.sh
/etc/profile.d/dotnet-cli-tools-bin-path.sh
/etc/profile.d/gawk.sh
/etc/profile.d/parrot.sh
/etc/profile.d/vte-2.91.sh
/etc/profile.d/Z97-byobu.sh
/home/midorino3142/.bashrc
/home/midorino3142/.cargo/env

$ echo exit | strace bash -l |& grep '^open' # + Some manual cleaning
/etc/profile
/etc/profile.d/bash_completion.sh
/etc/profile.d/debian-mate-default-settings_gtk-accessibility.sh
/etc/profile.d/dotnet-cli-tools-bin-path.sh
/etc/profile.d/gawk.sh
/etc/profile.d/parrot.sh
/etc/profile.d/vte-2.91.sh
/etc/profile.d/Z97-byobu.sh
/home/midorino3142/.profile
/home/midorino3142/.bashrc
/home/midorino3142/.cargo/env

Here, we note that method 2 is indeed superior for it correctly lists the files /etc/profile and /home/midorino3142/.profile, whereas method 1 did not. This is attended behaviour nonetheless, for these two files are automatically sourced by the system when Bash inits and are not referenced and sourced "manually" in any configuration files.

Some automatic cleaning for method 2:

$ echo exit | strace bash -l |& grep '^open' | grep -v "ENOENT" | awk '{ print $2 }' | tr -d ',' | tr -d '"'
(...)
/etc/bash.bash_logout

Note: ENOENT seems to be a keyword for case of file not found / loaded.