AI Agent Engineer

AI Agent Engineer

07 Pipelines, Redirection, and Command Composition

4 min read

07 Pipelines, Redirection, and Command Composition

Learning objectives

  • Pass one command’s output to another command.
  • Save output to a file.
  • Decide whether to continue based on the preceding command’s success or failure.
  • Distinguish stdin, stdout, stderr, and file descriptors, and verify intermediate pipeline failures.

Command overview

CommandPurposeMeaningBasic usageExample
|PipelinePasses standard output from the left command to the right commandcommand1 | command2ls -lah | grep -F '.log'
>Overwrite a fileWrites standard output to a file and replaces existing contentcommand > fileecho hello > a.txt
>>Append to a fileAppends standard output to a filecommand >> fileecho world >> a.txt
<Read input from a fileUses file contents as command inputcommand < filesort < names.txt
2>Redirect error outputWrites standard error to a filecommand 2> filefind . -name a 2> errors.txt
&&Run after successRuns the right command only when the left succeedscommand1 && command2mkdir test && cd test
||Run after failureRuns the right command only when the left failscommand1 || command2cd project || echo "directory does not exist"
teeDisplay and write simultaneouslyCopies stdin to stdout and a filecommand | tee fileprintf '%s\n' ok | tee result.txt

Common scenarios

Filter command output:

ls -lah -- . | grep -F '.log'
pgrep -a -u "$(id -u)" -f 'python3'

The first command demonstrates text filtering only. Process discovery uses pgrep so that ps | grep does not mistake grep’s own command line for a target process.

Save output to a file:

echo "hello" > message.txt
echo "world" >> message.txt

Save error output separately:

find . -name '*.conf' > find_results.txt 2> find_errors.txt

Continue only after success:

mkdir project && cd project

Print a message after failure:

cd project || echo "project directory does not exist"

Notes for beginners

  • > overwrites a file; use >> when original content must be retained.
  • A pipeline | passes output text, not the file itself.
  • 2> redirects only error output and does not affect normal output.
  • && is commonly used to avoid continuing when a preceding command failed.
  • Command composition is powerful. Preview with non-destructive commands before deleting or overwriting files.

The three standard streams and redirection order

Linux processes normally read stdin from file descriptor 0, write stdout to 1, and write stderr to 2. Redirections are processed left to right:

command_that_may_fail > combined.log 2>&1
command_that_may_fail 2>&1 > stdout-only.log

The two commands are not equivalent. The first makes fd 2 follow fd 1 after fd 1 points to the file; the second first points fd 2 at the original terminal and then moves fd 1 separately. The command name is only a signature explanation; do not execute it unchanged.

Redirection truncates first

sort data.txt > data.txt can truncate the input file before sort reads it. Write output to a new file, validate it, and then replace it in a controlled way.

Pipeline exit status

By default, Bash uses the last command’s status as the whole pipeline status:

false | true
pipeline_rc=$?
printf 'pipeline_rc=%s\n' "$pipeline_rc"

Assigning $? also overwrites PIPESTATUS, so rerun the pipeline and save the parts immediately:

false | true
parts=("${PIPESTATUS[@]}")
printf 'left=%s right=%s\n' "${parts[0]}" "${parts[1]}"

When Bash pipefail is enabled, a pipeline returns the status of its rightmost nonzero command; only a fully successful pipeline returns 0:

set -o pipefail
false | true
pipeline_rc=$?
printf 'pipeline_rc=%s\n' "$pipeline_rc"

PIPESTATUS and pipefail are Bash capabilities, not features of every POSIX Shell. A script also cannot rely only on set -e instead of explicit error handling because its behavior is context-sensitive in conditionals, subshells, and compound commands.

Controlled exercise

Create input.txt in a lab directory, write stdout, stderr, and combined output to three new files, then construct grep 'missing' input.txt | wc -l. Compare its default status, ${PIPESTATUS[@]}, and pipefail. Explain in the report why “0 lines” cannot prove that grep succeeded.

References

Backlinks