Building a robust CI pipeline for bash

Bash is one of the most popular scripting languages on servers and networked equipment, often regarded as "glue code" - no real application logic, but rather sticking two applications together. But under the hood, it is a programming language with its own problems and pitfalls and benefits heavily from a reliable automated CI pipeline.

Why build a CI pipeline for bash scripts?

Many teams view bash as glue to stick "real programs" together, assuming it is so simple that nothing could go wrong. Unfortunately, bash is significantly more complex and feature-rich than many users expect, including complexity and unexpected behavior.

Such issues are compounded by the fact bash fails silently by default, assuming default values for missing or misspelled variables, ignoring intermediate pipe failures, treating every value as a string and happily executing line by line in a script containing an obvious syntax error just one line down.

Bash is commonly used in highly important places: connecting one program's output to another, perhaps with light adjustments, backup automation scripts, scheduled cleanup jobs, ... the list goes on. Most of these will cause real damage or downtime to a production environment when a bash error makes it onto live environments.

Building a CI pipeline does not guarantee these problems go away, but it does catch a good number of them, keeps diffs minimal for faster debugging, and catches regressions early.

Static analysis

Since bash has no compiler or standard tooling, the open source shellcheck project has filled the gap as the de-facto standard static analysis linter for bash code. It can be configured in most code editors supporting language server protocol (LSP) by installing the Bash Language Server, offering real-time in-editor error catching while writing code.

It finds a lot of problems, from quoting issues, spacing problems in conditionals and assignments, to frequently misused commands like prematurely terminated -exec flags when using find. Many errors it finds are subtle in nature and would not stand out to developers unless they are intimately familiar with the bash language and are already familiar with the many pitfalls it contains.

For a reliable CI pipeline and workflow, you need to consider several points to make it robust:

1. Run it on every file ending in .sh (do not rely on --check-sourced, it misses scripts executed directly or via exec and eval)

2. Use the --shell=bash option to prevent false positives and portability warnings

3. Ignore supplied .shellcheckrc files with --norc and explicitly disable undesirable rules with --exclude=rule1,rule2,... instead (it will be abused by a lazy developer or under pressure of a deadline to cheat at pipeline success by degrading check quality, with side effects for the entire codebase)

4. Document that rules can be disabled individually on a per-line basis by using comments with the syntax # shellcheck disable=rule, so exceptions are easy to find, limited to a single line and reference an auditable author through git history

5. Explicitly set --severity=style in development environments / pre-commit hooks, but only block/fail commits or CI pipelines from issues found by --severity=warning

6. Check optional rules and enable any desirable ones with --enable=rule1,rule2,.... Consider --enable=all for development environments.

This sounds like a lot, but can be nicely tied into a short bash script:

find . -type f -name "*.sh" -exec \
  shellcheck --shell=bash \
    --norc \
    --enable=rule1,rule2,... \
    --exclude=rule1,rule2,... \
    --severity=style \
  {} +

This is just a template, adjust values for --enable, --exclude and --severity based on environment and project needs.

Code formatting

While code formatting doesn't contribute to catching errors directly, it does reduce noise in diff views between git commits when debugging or tracing logic regressions. Keeping differences between file versions focused on actual logic changes instead of meaningless formatting or whitespace minimizes developer overhead and saves time under pressure of error resolution. Remember bash scripts often serve vital purposes, so unlike most other software issues, bash errors typically result in immediate pressure to fix as quickly as possible.

The tool of choice will almost always be shfmt here, for its maturity and feature depth. What rules you set for formatting isn't nearly as important as just having a set of rules applied throughout the entire project. Formatting can be effectively invisible to developers by applying it on the fly with a git pre-commit

You can customize the formatting rules by storing a standardized .editorconfig alongside the source code, similar to this:

root = true

[*]
charset = utf-8
end_of_line = lf
insert_final_newline = true
trim_trailing_whitespace = true

[*.sh]
indent_style = space
indent_size = 2

Refer to editorConfig.org for a full set of options.

Development environments like pre-commit hooks should format files on the fly with

find . -type f -name "*.sh" -exec shfmt -ln=bash -w {} +

while CI pipelines should fail if any committed files are not properly formatted:

if find . -type f -name "*.sh" -exec shfmt -ln=bash -l {} + | grep -q .; then
    echo "Files are not properly formatted"
    exit 1
fi

Define execution behavior

By default, bash will fail quietly under many error conditions. To combat the worst of these cases, you will typically want to use a shebang referencing /usr/bin/env bash to select the interpreter, alongside common bash options set -euo pipefail, like this:

#!/usr/bin/env bash

set -euo pipefail

A CI pipeline should enforce that every bash file starts with these lines, to ensure developers can write code in known execution modes and deviations from this are deliberate, not accidental. Do not rely on inheritance from sourcing files, as other execution alternatives like exec or direct execution skip option inheritance.

Further checks

Depending on your project, you may or may not benefit from some additional pipeline features that won't apply to others. These include:

• Testing. The bats-core project is the most mature testing framework, supporting anything from unit testing to end-to-end testing approaches. It remains overkill for scripts and glue code, but can protect against regressions in larger cli tooling or automation projects.

• Typo checking. Misspelled strings may lead to subtle errors like checking for nonexistent associative array indexes, subtly wrong text substitution or outputs escaping grep checks. Use code-aware tools like cspell or typos.

• Custom compatibility checks, for example to make sure the script does not use features newer than your target bash version. These tests are often flaky regex matches, and considering all variations of flag ordering and multiline commands can be difficult. Avoid if possible.

Pre-commit hook

To make your life easier, here is a basic git pre-commit hook that you can put into .git/hooks/pre-commit to automate the three essential pipeline components:

#!/usr/bin/env bash

set -euo pipefail

# format all source files in-place and re-add to commit
find . -type f -name "*.sh" -exec shfmt -w {} +

# fail on shellcheck warnings and errors
find . -type f -name "*.sh" -exec \
  shellcheck --shell=bash \
    --norc \
    --enable=rule1,rule2,... \
    --exclude=rule1,rule2,... \
    --severity=style \
  {} +

# fail if required options are missing in any files
find . -type f -name "*.sh" -exec bash -c '
  for file do
    head -n 10 "$file" | grep -Fxq "#!/usr/bin/env bash" || {
      echo "Missing shebang in $file" >&2
      exit 1
    }
    head -n 10 "$file" | grep -Fxq "set -euo pipefail" || {
      echo "Missing strict mode in $file" >&2
      exit 1
    }
  done
' bash {} +

This should catch most issues without interrupting developers too much. Add documentation about enabling shellcheck with --severity=style to provide more hints from IDE/tooling during development.

Gitea/Github actions

On the server side, you can use a simple action yaml pipeline since most platforms support them, and unsupported ones like jenkins can use act as a compatibility emulator.

lint.yaml

name: bash linting

on:
  push:
  pull_request:

jobs:
  shell:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4

      - name: Install dependencies
        run: sudo apt-get update && sudo apt-get install -y shfmt shellcheck

      - name: Check formatting
        run: |
          output=$(find . -type f -name "*.sh" -exec shfmt -l {} +)
          if [ -n "$output" ]; then
            echo "shfmt issues found:"
            echo "$output"
            exit 1
          fi

      - name: Run shellcheck
        run: |
          find . -type f -name "*.sh" -exec \
            shellcheck --shell=bash \
              --norc \
              --enable=rule1,rule2,... \
              --exclude=rule1,rule2,... \
              --severity=style \
            {} +

      - name: Enforce shebang and strict mode
        run: |
          find . -type f -name "*.sh" -exec bash -c '
            for file do
              head -n 10 "$file" | grep -Fxq "#!/usr/bin/env bash" || {
                echo "Missing shebang in $file" >&2
                exit 1
              }
              head -n 10 "$file" | grep -Fxq "set -euo pipefail" || {
                echo "Missing strict mode in $file" >&2
                exit 1
              }
            done
          ' bash {} +

This will be a solid foundation for a bash CI pipeline. Extend it with testing or spell checking capabilities if beneficial for your project, and enjoy an implicit increase in script quality and reduced future on-call incidents.