VASST POSIX Shell and Bash Coding Style Guidelines
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

.. contents::

.. _shell_style_really:

Consider using Bash instead of the POSIX shell
----------------------------------------------

Bash provides many additional features for easier, more reliable, and more
secure programming. It is not a cure-all; almost all of the recommendations in
this document hold true for Bash programs, too.
However, arrays and ``[[ ]]`` conditionals that do not split variable values can be
a game changer.

There are some cases, e.g. when early on in the boot process, when it is not 100% certain
that Bash is available or installed at all.
In this case, POSIX shell programs should be written with additional care taken to
avoid the numerous pitfalls.

General
=======

File Layout
-----------

The general structure of a shell source file (both for library modules and
executable scripts) should be as follows:

.. code-block:: shell

  #!/bin/bash
  # (or, if the program can't use Bash, #!/bin/sh)
  #
  # A comment describing the purpose of the whole file.

  set -e
  # for Bash programs, also set -o pipefail

  usage()
  {
      ...
  }

  # also show_version(), show_features(), etc; see the below sections.

  do_something()
  {
     # One or more functions that perform the actual work of the program.
  }

  # If the program should parse command-line arguments:
  var1='default value'
  var2='another default value'
  unset noop qhelp qversion etc
  while getopts '...' o; do
     ...
  done
  # Handle --help, --features; shift OPTIND, etc; see the below sections.

  do_something
  if [ -n "$do_the_other_thing" ]; then
      do_something_else
  fi

Provide usage/help information
------------------------------

Most programs do some validation of command-line arguments, environment
variables, etc.
If something goes wrong, it is easy to call a ``usage`` function and then exit:

.. code-block:: shell

  usage()
  {
      cat <<'EOUSAGE'
  Usage:
      thistool [-fN] [-c /path/to/config.file] [-x exclude] pattern [file...]
      thistool [-fN] [-c /path/to/config.file] -m other-mode args
      thistool -h | -V | --help | --version | --features

      -c    specify the path to the configuration file to use
      -f    force mode; always do the thing, even if there will be trouble
      -h    display program usage information and exit
      -N    no-operation mode; display what would have been done
      -m    specify the operation mode (default: 'frob')
      -x    specify one or more patterns to exclude from frobbing
      -V    display program version information and exit
      --features
            display information about supported program features and exit

  Examples:
      thistool -N -c /path/to/config.txt -x everyone someone /etc/passwd
      thistool -f -m grok /etc/hostname
  EOUSAGE
  }

  ...
  if [ "$#" -lt 2 ]; then
      usage 1>&2
      exit 1
  fi

Consider providing a machine-readable "Features:" line
------------------------------------------------------

Sometimes another program needs to determine whether a program supports
a particular feature, specific file format/syntax, etc.
One of the best ways to handle this is for programs to declare what features
they support, possibly with semver-style major.minor version strings to
announce incompatible changes, etc.
In that case, the caller can either use a tool like the ``feature-check``
collection of libraries for different programming languages, or parse
the features line manually (mostly discouraged, but not outright forbidden).

This usually implies some level of processing command-line arguments using
e.g. ``getopts``.

Prefer ``getopts`` for processing command-line arguments
--------------------------------------------------------

The POSIX shell standardizes the ``getopts`` function with a syntax mostly
familiar to people used to passing single-letter flags and arguments to
programs.
Set some variables to default values, invoke ``getopts`` in a loop, and
store the results:

.. code-block:: shell

  show_version()
  {
      echo 'thistool 0.1.0'
  }

  show_features()
  {
      echo 'Features: thistool=0.1.0 frob=1.0 grok=1.1 config=2.1'
  }

  mode='frob'
  unset configpath exclude force noop qfeatures qhelp qversion
  while getopts 'c:fhm:Nx:V-:' o; do
      case "$o" in
          c)
              configpath="$OPTARG"
              ;;

          f)
              force=1
              ;;

          h)
              qhelp=1
              ;;

          m)
              mode="$OPTARG"
              ;;

          N)
              noop=1
              ;;

          x)
              # Mind the whitespace, this may need more careful processing.
              # Or, for Bash programs, declare and use an array instead.
              exclude="$exclude $OPTARG"
              ;;

          V)
              qversion=1
              ;;

          -)
              case "$OPTARG" in
                  features)
                      qfeatures=1
                      ;;

                  help)
                      qhelp=1
                      ;;

                  version)
                      qversion=1
                      ;;

                  *)
                      echo "Unrecognized long option '$OPTARG'" 1>&2
                      usage 1>&2
                      exit 1
                      ;;
              esac
              ;;

          *)
              usage 1>&2
              exit 1
              ;;
      esac
  done

  [ -z "$qversion" ] || show_version
  [ -z "$qfeatures" ] || show_features
  [ -z "$qhelp" ] || usage
  [ -z "$qversion$qfeatures$qhelp" ] || exit 0

  shift "$((OPTIND - 1))"
  if [ "$#" -lt 1 ]; then
      usage 1>&2
      exit 1
  fi

Note that it is not really straightforward to support long options with
arguments in this way.
Either use ``case ...  mode=*) mode="${OPTARG#*=}"``, avoid long options, or
write the program in a different language.

Avoid using ``$0`` in usage and version output
----------------------------------------------

The usage message should be formatted in a human-readable way, without
overly long lines and paths.
When writing it, the developer should usually know what name the program will
be invoked with; use that short name in the way the users will usually do that.

Use tabs, not spaces
--------------------

Use an eight-character tab indent.
Reflect this in the project's ``.editorconfig`` file.

The POSIX shell is called ``/bin/sh``, Bash is called ``/bin/bash``
-------------------------------------------------------------------

There are systems where the ``/bin`` and ``/usr/bin`` directories have been
merged.
Even before that, there were OSs and Linux distributions that packaged Bash in
such a way that it was available as both ``/bin/bash`` and ``/usr/bin/bash``.
Please ignore that.

The shell lives in the ``/bin`` directory; it will always be available there.
There are systems where ``sh`` and ``bash`` will NOT be available in ``/usr/bin``.

Use an empty string for a false value, and '1' for a true one
-------------------------------------------------------------

When storing a boolean value in a variable, using an empty string (instead of
e.g. "0") to represent a false value makes it possible to use the shell's
``[ -n "$var" ]`` and ``[ -z "$var" ]`` constructs to test for it (respectively,
Bash's ``[[ -n $var ]]`` and ``[[ -z $var ]]``).

Error messages, warnings, and diagnostic information go to the standard error stream
------------------------------------------------------------------------------------

Unless the shell program generates some kind of structured output that
has provisions for error messages, warnings, or diagnostic messages,
all such information must go to the standard error stream.

- people running the program in a terminal would expect that
- other programs trying to parse the program's output will be confused by
  free-form human-readable messages

Consider making diagnostic messages conditional on a command-line option or environment variable
------------------------------------------------------------------------------------------------

For diagnostic messages, something like the following may be used:

.. code-block:: shell

  debug()
  {
      [ -z "$v" ] || printf -- '%s\n' "$*" 1>&2
  }

  ...
  unset v
  while getopts '...v...' o; do
     case "$o" in
         ...
         v)
             v=1
             ;;
  ...

  debug "Using $tempd as a temporary directory"

Naming things
=============

Use ``snake_case`` for both function and variable names
-------------------------------------------------------

Do not use mixed case in function or variable names for shell programs.
Given that, separate words with underscores.

Use uppercase names for variables passed to or from other programs
------------------------------------------------------------------

It has been a convention in the Unix world for pretty much 50 years now that
variables passed from one program to another via the environment use uppercase
names.
Common examples include ``HOME``, ``SHELL``, ``SSH_CONNECTION``, ``LANG``, etc.

If the program looks for configuration values in the environment, please look
for variables named in uppercase.

Use lowercase names for the program's own variables
---------------------------------------------------

There are conflicting style recommendations on this topic.
One point of view is that it is much, much easier to figure out which variables are
local to the function or the program if they have lowercase names:

.. code-block:: shell

  do_something()
  {
      local name="$1"
      local count

      count="$(wc -l -- "$tempf" | awk '{ print $1 }')"
      printf -- '%s\t%s\n' "$name "$count"
  }

  ...
  unset tempf
  trap 'if [ -n "$tempf" ]; then rm -rf -- "$tempf"; fi' EXIT HUP INT QUIT TERM
  tempf="$(mktemp -t linecount.txt.XXXXXX)"

  ...
  do_something "$3"

Functions and variables
=======================

Put as much of the logic as possible in functions
-------------------------------------------------

Put as much of the program logic as possible in separate functions that
do one thing and do it well.

Avoid global variables if possible
----------------------------------

Pass variables as parameters to functions if possible.
There may be an exception for a handful of variables initialized from
command-line options or from the program's initial examination of its execution
environment.

Use local variables
-------------------

If a variable only needs to be used inside a single function, declare
it as local at the top of the function or immediately before its first use.

Always initialize local variables when declaring them
------------------------------------------------------

Even if the variable's value is not known yet (or must be obtained by
executing a command; see the "Error handling" section below), still
initialize the variable with an empty value or an invalid one.
Some shells allow uninitialized variables to "inherit" a value from
another variable with the same name that is currently in scope; this may
lead to weird side effects and bugs.

Quoting variables and expressions
=================================

ALWAYS, always, ALWAYS double-quote variable expansions!
--------------------------------------------------------

In the vast, vast majority of cases, when a variable is used, its value
should be treated as a single word by the shell, no matter whether
the variable contains any whitespace characters (space, tab, newline, etc).
Make a habit of always interpolating variables inside double-quoted
expressions so that a stray space in a command-line option supplied by
the user or in a value read from a configuration file does not make
the program run a command it was never meant to.

Exception: running another program with more than one variable argument
-----------------------------------------------------------------------

Sometimes a program needs to execute another program, pass on some
arguments specified on the command line, and prepend or append others.
Either use Bash arrays to do that, or use a separate function that
executes the other program and ``set -- ... "$@" ...`` to modify
the argument list. If the conditionally added arguments are very, very few,
another option is to use ``${varname:+-option "$arg"}`` instead.

.. code-block:: shell

  do_something()
  {
      local configpath="$1" mode="$2" noop="$3"
      shift 3
      # Now $1, $2, "$@" in general, contain the user-supplied arguments.

      ...
      # Add any more arguments if we need to...
      set -- -d "$topdir" "$@"
      if [ -n "$configpath" ]; then
          set -- -c "$configpath" "$@"
      fi
      if [ -n "$noop" ]; then
          set -- -N "$@"
      fi

      # The above may also be written as single-line complex commands:
      [ -z "$configpath" ] || set -- -c "$configpath" "$@"
      [ -z "$noop" ] || set -- -N "$@"

      # And now "$@" contains all the arguments that we need to run the other tool with.
      other-tool "$@"

      # Alternatively, the `set foo "$@"` construction above may be replaced by
      # conditionals on the line where the other program is invoked:
      other-tool -d "$topdir" ${configpath:+-c "$configpath"} ${noop:+-N} "$@"
  }

  do_something "$configpath" "$mode" "$noop" "$@"


In the extremely rare case when the arguments need to be read from
a configuration file or an environment variable, make as much effort as
possible to validate them, but in the end it may be acceptable to use
an unquoted variable expansion so that the shell passes more than
one word as an argument to the executed program.
However, this is indeed an extremely rare case; try to avoid it if at all
possible.

Put quotes around the longest expression possible
-------------------------------------------------

This piece of source code:

.. code-block:: shell

  scp -- "$filename" "buildvault:/temp/$cluster/$backuppath"

...is easier to read than this one:

.. code-block:: shell

  scp -- "$filename" buildvault:/temp/"$cluster"/"$backuppath"

Use single quotes for strings that do not contain variable interpolations
-------------------------------------------------------------------------

Do not use double quotes unless they are needed; this will also avoid
long sequences of backslashes escaping double quotes, special characters,
etc.

Validator utilities
===================

Use the shellcheck tool
-----------------------

There are only a very, very few issues flagged by shellcheck that
are overstated in importance.
In general, use the following as a ``.shellcheckrc`` file for the project:

.. code-block::

  shell=bash
  enable=all
  disable=SC2155

These configuration settings make the ``shellcheck`` tool go through all of
its checks except the one that enforces the sepration of declaring a variable and assigning it a value

In very few cases, individual shellcheck warnings and errors may be
overridden with a ``# shellcheck disable=SCfoo`` directive on
the preceding line.

In extremely few cases, individual shellcheck warnings and errors may be
overridden with a disable directive at the top of the file.
However, this should only be done when absolutely necessary.

Error handling
==============

Use the shell's errexit and pipefail modes
------------------------------------------

In most cases, ``set -e`` at the top of the file should be enough to
make sure that most of the commands invoked will lead to the program
(or the function) terminating on errors instead of blithely stomping
onwards in a messed-up environment.

For Bash programs, also set the ``pipefail`` option; this may eliminate
the need to use temporary files in some cases when passing data between
two programs.
However, bear in mind that this may still lead to problems if the second
program is executed and it receives incomplete data or no data at all on
its standard input stream.

Keep in mind that errexit mode may be disabled in some cases
------------------------------------------------------------

The shell's ``errexit`` setting is ignored if the shell considers that
the result of a command is checked in some way, e.g. it is part of
an ``if`` or ``while`` condition, a ``||`` or ``&&`` chain, etc.
In those cases, additional care must be taken to check for unsuccessful
operations and not proceed unaware.

Always test the error modes that the program checks for
-------------------------------------------------------

Make a habit of testing (either manually or using an automated
unit test or regression test tool (or a separate program)) that
the program will indeed detect and handle unsuccessful operations
correctly.

Use command substitution on a line separate from a variable declaration
-----------------------------------------------------------------------

When a local variable should contain the output of a command, initialize
the variable with an empty string in its declaration and then store
the command's result into it on a separate line.
Some shells may ignore the failure of a command executed within
a ``local`` statement.

.. code-block:: shell

  do_something()
  {
      local topdir=''
      
      topdir="$(get_topdir)"
  }

Do not reinvent the wheel
=========================

Use the Programmer's Toolbox
----------------------------

The standard Unix set of text- and file-handling tools, usually known to
Linux users as the ``coreutils``, ``findutils``, and at some point also
``textutils`` before they were merged into the ``coreutils`` package, will be
available on the systems that the shell program runs on.
Make use of them instead of reinventing them poorly by reading from files,
analyzing their contents line by line, etc.

Use the shell's own constructs
------------------------------

Get to know the shell's various substitution modes -
`for the POSIX shell <http://gondor.apana.org.au/~herbert/dash/>`_ and
`some Bash extensions <https://www.gnu.org/software/bash/manual/html_node/Shell-Parameter-Expansion.html>`_ -
e.g.:

- remove the smallest prefix pattern: ``${var#*=}``
- remove the largest suffix pattern: ``${var%%=*}``
- perform simple arithmetic: ``i="$((i + 1))"``
  (note that there is no need for ``$i`` within the expression: the shell can recognize
  a variable name and substitute its value)
- provide a default value for uninitialized variables: ``: "${CONFGET:=confget}"``
  (please note that the ``:`` at the start is not a typo, but a no-op command
  used for evaluating its arguments)
- pass arguments to a program if a variable is defined: ``${config:+-c "$config"}``
- pass a default value of an argument if not specified: ``${FILENAME:-/etc/hostname}``

Do not be afraid of using temporary files
-----------------------------------------

Even though most shells provide some support for multiline variable values,
sometimes storing a program's output into a temporary file and using
the standard text processing tools is more straightforward.
However, make sure to clean up afterwards!

.. code-block:: shell

  unset tempf
  trap 'if [ -n "$tempf" ]; then rm -rf -- "$tempf"; fi' EXIT HUP INT QUIT TERM
  tempf="$(mktemp -t indices.txt.XXXXXX)"

Sometimes using a temporary directory is better, although some care should be
taken with that, since the ``mktemp`` utility's support for temporary directories
has not been standardized:

.. code-block:: shell

  unset tempd
  trap 'if [ -n "$tempd" ]; then rm -rf -- "$tempd"; fi' EXIT HUP INT QUIT TERM
  tempd="$(mktemp -d -t frob.XXXXXX)"

When using ``trap`` to clean up, please note that a later ``trap`` invocation will
replace the previous one.
There is no portable, supported way to append actions to a trap.
If a program needs to set different clean-up routines for different cases, either
run a function in the trap handler and have it checked global variables for
empty or non-empty values, or, you guessed it, rewrite the program in
a different language.

Use ``find`` instead of ``ls``
------------------------------

The ``ls`` program has never been designed for automated use.
The ``find`` tool, on the other hand, has been designed to provide output for
other programs right from the start.
The benefits of using ``find`` instead of ``ls`` include, but are not limited to:

- avoiding problems with ``ls`` formatting some filenames in special ways
- complete control over the way the filename (and any additional data) is formatted
- more control over file selection using the predicates supported by ``find`` as
  opposed to running ``grep`` over the output of ``ls``
- avoiding problems with filenames containing whitespace characters

Use Bash's features
----------------------------------

Use Bash features that are available:

- ``[[ ]]`` conditionals that do not split variable values on whitespace
- ``=~`` in ``[[ ]]`` conditionals for regular expression matching
- arrays of values
- ``$RANDOM`` that may also be used as e.g. ``$((RANDOM % 10))``
- ``set -o pipefail`` for slightly more robust error handling (but see the comment
  in the "errexit and pipefail" section above)

Use the -e option when passing patterns to grep, sed, etc
---------------------------------------------------------

There are several reasons:

- security: a pattern starting with a dash will not be interpreted as
  a command-line option
- clarity: it is obvious to the reader that an arbitrarily complex expression at
  that point is a pattern and not e.g. a filename constructed from several variables
- uniformity: the same syntax is used when passing a single pattern and more than
  one of them, e.g. ``grep -Eqe first -e second -e third``

Use grep's -q and -x options as appropriate
-------------------------------------------

When looking for whole lines in a file, use ``grep -Fxe 'whole line'`` or
``grep -Exe 'pattern for the whole line'``.

When using ``grep`` in an ``if`` statement, avoid unnecessary output to
the standard error stream (and unnecessary redirections to /dev/null that
will still cause the ``grep`` tool to "output" the lines) by using
``grep -Eqe '...'`` or ``grep -Fqe '...'``.

Use text utilities in the optimal way
-------------------------------------

- grep + sed = sed: ``sed -ne '/foo/ { s/bar/baz/; p; }'``
- grep + awk = awk: ``awk -V var=value '$1 == value { print $3 }'``

.. _shell_style_security:

Security
========

This... is woefully inadequate. More will come later.

If it seems that these guidelines for making a shell program at least
somewhat reliable and resistant to at least the most basic security
attacks are too many and too complicated to follow, please refer to
the :ref:`shell_style_really` section above.

ALWAYS, always, ALWAYS double-quote variable interpolations!
------------------------------------------------------------

This may have been mentioned in previous sections, but there are really
EXTREMELY few cases when a variable must be used outside of a double-quoted
expression.
Please make a habit of that, so that it does not get forgotten in a case when
it will really, really matter.

Use two dashes as a delimiter between optional and positional arguments
-----------------------------------------------------------------------

Some programs (especially ones coming from or influenced by the GNU project)
will accept optional arguments starting with a dash even after they have
encountered a positional argument on the command line (e.g. ``ls /bin -l``).
Since many commands are invoked with user-supplied input as arguments, if
a variable starts with a dash, the command may misinterpret it and do
the wrong thing (sometimes resulting in security issues, not only reliability ones).
This is especially true when dealing with filenames, program names, etc, since
nothing prevents anybody from using a dash to start a valid filename that will
later be passed to a program.

Almost all programs, commands, and tools honor the ``--`` special "argument" that
means "everything that comes later is a positional argument, do not attempt to
interpret it as flags or options":

.. code-block:: shell

  get_topdir()
  {
      local prog=''

      prog="$(readlink -f -- "$0")"

      # If the program itself is in the top-level directory:
      dirname -- "$prog"

      # If the program is one directory deeper:
      local progdir=''

      progdir="$(dirname -- "$prog")"
      dirname -- "$progdir"
  }

Use ``printf`` instead of ``echo``, especially for strings starting with a variable
-----------------------------------------------------------------------------------

Most implementations of the ``echo`` command (which may be a shell built-in)
support some command-line options, e.g. ``-n`` for not outputting a newline character
at the end, or ``-e`` for interpreting escape sequences such as "\\n", "\\t", etc.
Unfortunately that means that ``echo`` will look at the first characters of
the string passed to it, and if they are ``-e`` or ``-n`` or something similar,
it may NOT output them directly.
With many shells, including Bash, this can sometimes happen even if the variable is
placed in double quotes, since that kind of substitution happens long before
the ``echo`` builtin gets to analyze its input parameters.

When outputting a message that must start with a variable, use ``printf`` instead:

.. code-block:: shell

  printf -- '%s or something\n' "$varname"

Use temporary files (or directories) instead of files with predictable names
----------------------------------------------------------------------------

When storing some data in a file for easier processing with grep, sed, etc.,
do NOT put it in e.g. a file called "data.txt" in the current directory or
in /tmp; this opens up several avenues of attack or misbehavior:

- symlink attacks: if the program runs with elevated privileges, an attacker may
  create a symlink with that name, so that the program will overwrite another
  file that the attacker may not have access to
- simultaneous execution: if two people run the same program at the same time,
  things will get a little bit confusing when both attempt to store their data in
  the same file and then read it back
- files with common names: somebody may actually want to store their own data in
  a file called e.g. 'data.txt' or 'input-files.csv' or something similar;
  it might be a bit rude if an unrelated program overwrites it

Do NOT write parsers for complicated input data (JSON, HTML, INI-style files, etc)
----------------------------------------------------------------------------------

Do not attempt to write a shell parser for JSON, HTML, INI-style files, or other
structured data!

DO NOT attempt to write a shell parser for JSON, HTML, INI-style files, or other
structured data!

Use one of the ready-made and well-tested tools that can parse
files in certain formats (e.g. ``jq`` for JSON data, ``xq`` or ``xmlstarlet`` for XML data,
``confget`` for INI-style files) and output parts of them in a form suitable for
consumption by shell programs.

DO NOT attempt to write a shell parser for JSON, HTML, INI-style files, or other
structured data!

DO NOT SOURCE INI-STYLE FILES!
------------------------------

If a file consists mostly of ``variable=value`` pairs and ``# comment`` lines,
it may seem that it is easy enough to source it into the shell.
This can lead to many security problems or simply failure modes if the variable
names or the value names contain special characters that will be misinterpreted by
the shell; the most obvious example is ``$(...)`` command substitution, but
even "just" variable interpolations may have very much undesired side effects.

Use ``confget`` with its ``-S`` (escape output for the shell) option and,
preferably, also the prefix and suffix variable name options.

Pass variables to tools that support that, e.g. ``awk`` or ``jq``
-----------------------------------------------------------------

When invoking a tool that has its own command language, it may be tempting to
pass it a double-quoted string containing variable interpolations as the program
to execute.
This, however, may lead to all sorts of problems if the variables contain special
characters that are interpreted in unexpected ways by the tool.

If the tool supports a syntax for specifying variables on the command line,
like ``awk -V var="$value"`` or ``jq --arg var "$value"``, please use that.

Use null-terminated strings for communication between programs if possible
--------------------------------------------------------------------------

If processing one program's output via another, make sure to avoid problems
related to filename containing whitespace characters (space, tab, etc) by
telling the programs to use a null character as a terminator, for the output or
the input respectively.
With some programs this is done using a ``-0`` or a ``-z`` command-line option:

.. code-block:: shell

  git ls-files -z | xargs -r -0 -- tar cf "$tempd/source.tar" --

However, please note that not all tools may support such options.

Use ``find -exec`` instead of ``find | xargs``
----------------------------------------------

The historical use of ``find ... -print0 | xargs -0 -r -- ...`` is non-standard;
it may fail on some systems.
Additionally, it may also fail if ``find`` fails to actually find any matching files,
since the ``-r`` option to ``xargs`` is also non-standard.

The preferred way to run a command with the names of one or more matching files as
arguments is as follows:

.. code-block:: shell

  # Run a command on each file separately
  find ... -exec command -args '{}' more args ';'

  # Run a command, passing as many filenames as will fit, all at once
  find ... -exec command -args '{}' more args +

The ``find`` tool will pass each filename as a separate command-line argument to
the executed program without making a mess with whitespace characters.