Pipes, Redirects, File Descriptors

Every Unix process has three default file descriptors: 0 (stdin), 1 (stdout), 2 (stderr). Pipes and redirects rewire them. Knowing the syntax fluently is the difference between “I have an idea” and “I’m done writing the script.”

The three default FDs

Errors and warnings should go to stderr; data goes to stdout. That way cmd > out.log 2> err.log cleanly separates them, and cmd | next only pipes the data.

Output redirects

cmd > file               # stdout to file (overwrite)
cmd >> file              # stdout to file (append)
cmd 2> file              # stderr to file
cmd 2>> file             # stderr to file (append)
cmd > out 2> err         # both, separate files
cmd > /dev/null          # discard stdout
cmd 2> /dev/null         # discard stderr
cmd &> /dev/null         # discard both (bash shorthand)
cmd > /dev/null 2>&1     # POSIX equivalent of &> — both to /dev/null
cmd 2>&1 > file          # stdout → file, stderr → terminal (NOT what you usually want)
cmd > file 2>&1          # stdout → file, stderr → file (correct order)

The order of > file 2>&1 matters. Read it left to right:

1. > file — stdout now points to file.
2. 2>&1 — stderr now points to wherever stdout points (which is file).

Reverse order:

1. 2>&1 — stderr now points to wherever stdout points (terminal).
2. > file — stdout now points to file (but stderr still points to terminal — too late).

Memorize: > file 2>&1 is the right order.

The bash shorthand &>file and &>>file (append) avoid the gotcha:

cmd &> all.log
cmd &>> all.log

Input redirects

cmd < input              # stdin from file
cmd < <(other_cmd)       # process substitution — stdin from a command's output
cmd <<< "string"         # here-string — stdin from a literal string
cmd <<EOF                # heredoc
multi-line
input
EOF

wc -l < /etc/passwd                      # 49 (just the number, no filename)
grep "term" < log.txt
sort <<< "$multi_line_string"

<<< (here-string) is convenient for one-line stdin:

read -r year month day <<< "2024 01 15"
echo "$year $month $day"                  # 2024 01 15

Pipes

cmd1 | cmd2                              # cmd1's stdout → cmd2's stdin
cmd1 | cmd2 | cmd3                       # chain
cmd1 |& cmd2                             # bash shorthand: stdout AND stderr → cmd2
cmd1 2>&1 | cmd2                         # POSIX equivalent

Pipes only carry stdout by default. To pipe stderr too:

make 2>&1 | grep -i error                # catches errors in make's stderr
make |& grep -i error                    # bash shorthand

tee — split output to multiple destinations

cmd | tee output.log                     # to file AND to terminal
cmd | tee -a output.log                  # append (don't overwrite)
cmd | tee output.log | next_cmd          # pipe continues
cmd | tee >(grep error > errors.log)     # tee to a process substitution

tee is for “save this AND keep using it” — typical for long-running commands where you want the output in a log and on-screen.

For commands that need root to write the destination:

echo "127.0.0.1 example.com" | sudo tee -a /etc/hosts

sudo echo ... > /etc/hosts doesn’t work because the shell does the redirect (as your user) before sudo runs. sudo tee is the workaround.

Pipe failure semantics

In bash by default, a pipeline’s exit status is the last command’s status:

false | true       # exit 0 (true succeeded)
echo $?            # 0

This silently masks errors in earlier pipeline stages. Fix:

set -o pipefail    # pipeline fails if ANY command fails
false | true       # exit 1 now

Always use set -o pipefail in scripts. See 06_exit_codes_strict_mode.md.

Process substitution

diff <(sort file1) <(sort file2)         # diff two sorted versions without temp files
comm -12 <(sort a.txt) <(sort b.txt)     # lines in both

<(cmd) makes the command’s stdout appear as a file path (typically /dev/fd/63). Bash-specific.

>(cmd) is the opposite — write to a path, the bytes go to the command’s stdin:

cmd | tee >(gzip > out.gz) > out.txt     # tee to both gzip AND a plain file

Custom file descriptors

You can open arbitrary FDs:

exec 3> /tmp/myfd                        # open FD 3 for writing
echo "hello" >&3                          # write to FD 3
exec 3>&-                                 # close FD 3

Useful for keeping a log file open across multiple commands without reopening:

exec 3>>/var/log/myscript.log
echo "starting" >&3
do_thing >&3 2>&3
echo "done" >&3
exec 3>&-

Or to swap stdout temporarily:

exec 4>&1                                 # save current stdout to FD 4
exec > /tmp/redirected.log                # redirect stdout to file
echo "this goes to log"
exec >&4                                  # restore stdout from FD 4
exec 4>&-                                 # close the saved FD
echo "this goes to terminal"

/dev/null and /dev/stdin etc.

cat file > /dev/null               # measure read time without printing
dd if=/dev/zero of=test.bin bs=1M count=100   # 100 MB file of zeros
head -c 1000 /dev/urandom | base64           # random base64

xargs and stdin → arguments

Many tools take args, not stdin. xargs bridges the two:

find . -name "*.tmp" | xargs rm                  # rm each match
find . -name "*.tmp" -print0 | xargs -0 rm       # safe with spaces
echo "1 2 3" | xargs -n 1 echo                   # one arg per call
seq 10 | xargs -n 1 -P 4 ./worker.sh             # 4 parallel workers

xargs -0 paired with find -print0 handles filenames with spaces, newlines, etc. Always use this combo for find-driven loops.

Pipe gotchas

“While read” subshell

count=0
seq 5 | while read -r n; do
    count=$((count + 1))
done
echo "$count"        # 0 (!) — the while ran in a subshell

Each side of a pipe runs in its own subshell. Variable changes inside while don’t leak out. Fix:

count=0
while read -r n; do
    count=$((count + 1))
done < <(seq 5)      # process substitution avoids the subshell
echo "$count"        # 5

Or use shopt -s lastpipe (bash 4.2+).

Buffering

grep, sed, awk line-buffer when stdout is a terminal but block-buffer when stdout is a pipe. So tail -f log | grep error may not show anything for a while.

tail -f log | grep --line-buffered error
tail -f log | stdbuf -oL -eL grep error
tail -f log | unbuffer grep error           # from `expect` package

Common interview confusions

• “> and >> are the same.” — > overwrites; >> appends.
• “2>&1 > file redirects both.” — wrong order. > file 2>&1 is correct: stdout first, then stderr follows it.
• “Pipes pass stderr too.” — only stdout. Use 2>&1 | next or |& for both.
• “while read modifies variables you can use after.” — only without a pipe. With a pipe, the loop runs in a subshell and changes vanish.

Interview angle

• “What’s the difference between FDs 1 and 2?” — 1 is stdout (data); 2 is stderr (errors/warnings). Separating them lets you redirect logs cleanly.
• “How do you redirect both stdout and stderr to a file?” — cmd > file 2>&1 (POSIX) or cmd &> file (bash shorthand). Order matters in the POSIX form.
• “What does set -o pipefail do?” — without it, a pipeline’s exit status is the last command’s only — earlier failures are masked. With it, the pipeline fails if any command fails.
• “How do you save command output to a file AND see it on screen?” — cmd | tee file.log. Use -a to append.
• “Why doesn’t sudo echo > /etc/hosts work?” — the shell does the redirect as the original (non-root) user before sudo runs. Use echo ... | sudo tee -a /etc/hosts.
• “You’re piping find to rm and it breaks on filenames with spaces — how do you fix it?” — find . ... -print0 | xargs -0 rm. Uses NUL bytes as separators, which can’t appear in filenames.
• “Why might tail -f log | grep error not show anything?” — grep block-buffers when stdout is a pipe. Use --line-buffered (or stdbuf -oL) to flush per-line.