backend / linux bash / 10_signals_traps.md

Signals and Traps

6 interview angles 7 min read source

Signals and Traps

Signals are how the kernel and processes tell each other “something happened.” For Python developers the relevant ones are: SIGTERM (please shut down), SIGKILL (force shutdown), SIGINT (Ctrl-C), SIGHUP (terminal closed / often “reload”), SIGUSR1/USR2 (app-defined). Knowing how to send and catch them is essential for graceful shutdown.

The signals you need

Name Number Meaning Catchable? Default action
SIGHUP 1 terminal hangup; convention: reload config yes terminate
SIGINT 2 interrupt (Ctrl-C) yes terminate
SIGQUIT 3 quit + dump core (Ctrl-) yes terminate + core
SIGKILL 9 force kill NO terminate
SIGTERM 15 polite termination yes terminate
SIGSTOP 19 pause NO stop
SIGTSTP 20 suspend (Ctrl-Z) yes stop
SIGCONT 18 continue (resume after stop) yes resume
SIGUSR1 10 user-defined #1 yes terminate
SIGUSR2 12 user-defined #2 yes terminate
SIGCHLD 17 child process state changed yes ignored
SIGPIPE 13 wrote to a closed pipe yes terminate

Numbers vary by architecture; names are portable. Use names (kill -TERM) not numbers.

SIGKILL vs SIGTERM — the most asked

SIGTERM: “please shut down.” The process can:

  • Catch the signal.
  • Run cleanup (close connections, flush buffers, persist state).
  • Exit when ready.

SIGKILL: “die now.” The kernel terminates the process immediately. The process cannot:

  • Catch it.
  • Clean up.
  • Do anything.

kill <pid> sends SIGTERM. kill -9 <pid> sends SIGKILL. Use TERM first, give the process a few seconds, then KILL only if it’s hung.

SIGSTOP vs SIGTSTP

Both pause a process. Difference:

  • SIGSTOP (signal 19): uncatchable. The kernel stops the process immediately. Resume with SIGCONT.
  • SIGTSTP (signal 20, sent by Ctrl-Z): catchable. Apps that want graceful suspend can handle it.

Useful: kill -STOP <pid> to freeze a runaway process so you can investigate, then kill -CONT <pid> to resume (or kill -KILL to terminate).

sending signals

kill 1234                       # SIGTERM (default)
kill -TERM 1234                 # explicit
kill -15 1234                   # by number
kill -9 1234                    # SIGKILL
kill -HUP 1234                  # SIGHUP (often = reload)
kill -USR1 1234                 # send USR1
kill -l                         # list all signal names
killall gunicorn                # by name
pkill -HUP nginx                # by pattern

To kill all processes of a user (don’t run on yourself):

pkill -u alice

What signals get sent automatically

Action Signal sent
Ctrl-C in terminal SIGINT to foreground process group
Ctrl-Z in terminal SIGTSTP to foreground process group
Ctrl-\ in terminal SIGQUIT to foreground process group
Closing terminal SIGHUP to processes attached to that TTY
Process writes to a closed pipe SIGPIPE to the writer
Child exits SIGCHLD to parent
OOM killer fires SIGKILL to chosen victim
systemctl stop SIGTERM, then SIGKILL after timeout

Trap — handling signals in bash

#!/usr/bin/env bash
set -euo pipefail

cleanup() {
    echo "cleaning up..."
    rm -rf "$work_dir"
}

trap cleanup EXIT                       # always run on exit
trap 'echo "interrupted"; exit 130' INT  # Ctrl-C
trap 'echo "terminated"; cleanup; exit 143' TERM
Trap pseudo-signal When
EXIT script exits, any reason
ERR any command fails (with set -e)
DEBUG before every command (debugging)
RETURN when a function returns

trap - INT resets the INT handler to default.

The cleanup pattern

#!/usr/bin/env bash
set -euo pipefail

work_dir=$(mktemp -d)
trap 'rm -rf "$work_dir"' EXIT

# ... do work in $work_dir ...
# Even on Ctrl-C, error, or normal exit, work_dir is cleaned up

This is the canonical “make a scratch directory, guarantee cleanup” pattern. Combine with mktemp for safety.

Graceful shutdown in Python

A real Python service should catch SIGTERM:

import signal
import sys

def graceful_shutdown(signum, frame):
    print("shutdown requested, flushing...")
    flush_buffers()
    close_connections()
    sys.exit(0)

signal.signal(signal.SIGTERM, graceful_shutdown)
signal.signal(signal.SIGINT, graceful_shutdown)

Most frameworks (Django, Flask, FastAPI via uvicorn/gunicorn) handle this for you. Check your worker class’s docs:

  • Gunicorn (sync): SIGTERM = graceful shutdown (finish in-flight requests, then exit). SIGINT = same. SIGQUIT = quick shutdown. SIGKILL = die now.
  • Uvicorn: similar; SIGTERM gives --timeout-graceful-shutdown to drain.
  • Celery: SIGTERM = warm shutdown; SIGQUIT = cold.

Docker / Kubernetes signal flow

When docker stop or kubectl delete pod:

  1. SIGTERM sent to PID 1 in the container.
  2. Wait for --timeout (default 10s in Docker, terminationGracePeriodSeconds in k8s, default 30s).
  3. SIGKILL if still running.

Two common bugs:

PID 1 problem

CMD ./start.sh

If start.sh is a shell script that execs the real binary, the shell gets the signal but doesn’t forward it. Two fixes:

  1. exec in the script: exec gunicorn ... (replaces shell with the binary as PID 1).
  2. Init system: tini or dumb-init as PID 1 forwards signals correctly.
ENTRYPOINT ["tini", "--"]
CMD ["python", "app.py"]

App ignores SIGTERM

If the app catches SIGTERM and does nothing (or doesn’t catch it), the 10/30-second graceful period is wasted, then SIGKILL kills it mid-request. Symptoms: dropped connections during deploys.

Always implement graceful shutdown in production services.

SIGPIPE — the silent killer

When you cmd1 | cmd2 and cmd2 exits early, cmd1 writing to the closed pipe gets SIGPIPE. Default action: terminate cmd1.

This is usually fine (yes | head -5 works because of SIGPIPE), but in scripts can produce surprising failures:

generate_huge_output | head -5
# generate_huge_output gets SIGPIPE after head closes — exits 141 (128+13)

With set -e -o pipefail, this can fail your script. Workarounds:

generate_huge_output | head -5 || true
# or
{ generate_huge_output; } | head -5

Process groups and pgid

Signals can be sent to a process group (all processes started together):

kill -- -1234                     # negative PID = send to process group 1234

Used by shells when you Ctrl-C — the signal goes to the foreground process group, including children.

Common pitfalls

  • Using kill -9 first — skips cleanup. Try TERM, wait 5–10s, then KILL.
  • Trapping EXIT but using exit inside the trap — fine; the trap doesn’t re-run on exit triggered by itself.
  • Catching SIGKILL or SIGSTOP — impossible. The kernel handles them directly.
  • Container app not handling SIGTERMdocker stop always feels like 10 seconds, then nukes you. Implement graceful shutdown.
  • Shell script as PID 1 without exec — signals get swallowed. Use exec or tini.

Common interview confusions

  • “SIGTERM and SIGKILL are the same.” — TERM is polite; the process can clean up. KILL is forced; the process can’t react.
  • “You can catch SIGKILL with the right code.” — no. Uncatchable, by kernel design.
  • kill deletes a process.” — sends a signal. Default signal is SIGTERM. The process exits if it doesn’t have a handler.
  • “Ctrl-C kills the program.” — sends SIGINT. Default is to terminate, but the program can catch and ignore (some interactive programs do).

Interview angle

  • “What’s the difference between SIGTERM and SIGKILL?” — TERM is catchable (process can clean up); KILL is uncatchable, immediate. Always TERM first; KILL only if it doesn’t respond.
  • “Why can’t you catch SIGKILL?” — kernel handles it directly; no signal handler runs. Same with SIGSTOP. By design — there has to be a guaranteed way to terminate any process.
  • “How does docker stop work?” — SIGTERM to PID 1; wait 10 seconds (default); SIGKILL. App needs to handle SIGTERM gracefully.
  • “Why do you need tini in a Docker image?” — PID 1 has special signal-handling rules. A typical Python interpreter or shell script as PID 1 may not properly reap zombies or forward signals. tini is a tiny init that does both.
  • “What’s the trap pattern for cleaning up a temp directory?”work=$(mktemp -d); trap 'rm -rf "$work"' EXIT. Runs cleanup on any exit (success, error, signal).
  • “What’s SIGHUP traditionally used for?” — terminal hangup (close). By convention many daemons treat it as “reload config” (nginx, postfix). Not enforced — the daemon has to catch and interpret it.