Welcome to part 2 of our 3-part series of “Unofficial Bash Strict Mode”. This post is about the notable behaviours of pipefail and how to overcome them. Will also refer to the Bash reference manual, where you will find more useful tips about Bash.
犀利士 >In our previous post we bloged on Bash Strict Mode and the errexit and it’s core functions like how it can improve your code behaviour by setting the errexit flag & how it places output on the standard error stream stderr and more. You can find it here. In this post, we will introduce you to pipefail and it’s noticeable behaviours.
You are trying to do this:
#!/usr/bin/env bash
set -e
non_existent_cmd | another_non_existent_cmd | cat
echo "Hello"
This would run just fine !
Unfortunately, errexit is not enough to save us here. As we recall from the manual:
The shell does not exit if the command that fails is: (…)
4. any command in a pipeline but the last, or if the command’s return status is being inverted with !.
The exit status of a pipeline is the exit status of the last command in the pipeline.
Let’s set pipefail:
If the pipefail is enabled, the pipeline’s return status is the value of the last (rightmost) command to exit with a non-zero status, or zero if all commands exit successfully.
In other words, it will only return 0 if all parts of the pipeline return 0. As opposed to errexit, pipefail can only be set by its full form:
As opposed to errexit, pipefail can only be set by its full form:
set -o pipefail
Let’s fix the example from before:
#!/usr/bin/env bash
set -eo pipefail
non_existent_cmd | another_non_existent_cmd | cat
echo "Hello"
Both behaviours are verified by the following Bats test:
#!/usr/bin/env bats
load '../../../node_modules/bats-support/load'
load '../../../node_modules/bats-assert/load'
@test "runs fine since 'pipefail' is not set" {
run "$BATS_TEST_DIRNAME/pipefail_first.sh"
[ "$status" -eq 0 ]
[ "${lines[2]}" == 'Hello' ]
}
@test "fails since 'pipefail' is set" {
run "$BATS_TEST_DIRNAME/pipefail_first_correct.sh"
[ "$status" -ne 0 ]
[ "${lines[2]}" != 'Hello' ]
}
#!/usr/bin/env bash
set -eo pipefail
cat non_existing_file | xargs curl -qs
Cat’s exit code is 1 for when the file doesn’t exist. And xarg’s exit code is 123 “if any invocation of the command exited with status 1-2”. Obviously, both are broken, but what exit code do we get here?
The answer is 123, which is not ideal.
Our recommendation is to simply break it down into different instructions:
#!/usr/bin/env bash
set -eo pipefail
contents=$(cat non_existing_file)
curl -qs "$contents"
This behaviour can be confirmed by the following Bats test:
#!/usr/bin/env bats
load '../../../node_modules/bats-support/load'
load '../../../node_modules/bats-assert/load'
@test "returns exit code of xargs" {
run "$BATS_TEST_DIRNAME/pipefail_quirk_1.sh"
[ "$status" -eq 123 ]
}
@test "returns exit code of cat" {
run "$BATS_TEST_DIRNAME/pipefail_quirk_1_correct.sh"
[ "$status" -eq 1 ]
}
#!/usr/bin/env bash
set -eo pipefail
function all_hosts() {
echo 'host-1
host-2
host-a
host-b'
}
function remove_hosts() {
hosts=$(all_hosts | tr 'n' ' ')
whitelist="$1"
echo "
Removing hosts: $hosts
Whitelist: '$whitelist'
"
# Imagine we are passing those two parameters
# To another command
}
cat non_existent_whitelist_file | remove_hosts
In this example, we’re loading a whitelist file, feeding it to another command ( implemented here as a function), and that passes it to yet another service (e.g., a CLI tool). Even though the file doesn’t exist, the pipeline doesn’t fail. This end up passing an empty string to remove_hosts, which could have catastrophic effects ( deleting more than you expect)!.
Ideally, you’d want to fail as soon as possible. The best way to do that is to break it down into more instructions and just be more careful.
#!/usr/bin/env bash
set - eo pipefail
function all_hosts() {
echo 'host-1
host-2
host-a
host-b'
}
function remove_hosts() {
hosts=$(all_hosts | tr 'n' ' ')
whitelist=$1"
echo "
Removing hosts:
$hosts
Whitelist:
'$whitelist'
"
# Imagine we are passing those two parameters
# To another command
}
readonly local whitelist_file="non_existent_whitelist_file"
if [ ! -f "$whitelist_file" ]; then
echo "Whitelist file does not exist"
exit 1
fi
cat "$whitelist_file" | remove_hosts
As always, this behaviour is described by the following Bats file:
#!/usr/bin/env bats
load '../../../node_modules/bats-support/load'
load '../../../node_modules/bats-assert/load'
@test "runs fine even though file does not exist" {
run "$BATS_TEST_DIRNAME/pipefail_quirk_2.sh"
[ "$status" -ne 0 ]
[ "${lines[2]}" == "Whitelist: ''" ]
}
@test "fails since we verify file presence" {
run "$BATS_TEST_DIRNAME/pipefail_quirk_2_correct.sh"
[ "$status" -eq 1 ]
{ "$output" == "Whitelist file does not exist" ]
}
Check out some more Examples of why pipefail is really important to use.
Other parts in this blog series:
Part 1 – errexit
Part 3 – Coming soon!
November 26th, 2020
by Robert Golabek in Technology
⟵ Back
See more:
December 10th, 2021
Cloud Composer – Terraform Deploymentby Patryk Golabek in Data-Driven, Technology
December 2nd, 2021
Provision Kubernetes: Securing Virtual MachinesAugust 6th, 2023
The Critical Need for Application Modernization in SMEs