Process Management

Every running program is a process with a PID, a parent (PPID), an owner (UID), and a state. Bash gives you tools to inspect, background, kill, and monitor them. The interview questions are: “what’s the difference between & and nohup?”, “how do you find what’s using a port?”, and “how do you kill a hung process?”

ps — what’s running

ps                                  # YOUR processes in this terminal
ps aux                              # everyone's processes (BSD-style)
ps -ef                              # everyone's processes (sysV-style)
ps aux | grep gunicorn              # filter (use pgrep instead)
ps -o pid,ppid,user,pcpu,pmem,cmd   # custom columns
ps --sort=-pcpu | head              # top by CPU
ps --sort=-rss | head               # top by memory (RSS)

The aux/-ef columns:

pgrep / pkill

Targeted version of ps | grep:

pgrep gunicorn                      # PIDs of processes matching "gunicorn"
pgrep -f "gunicorn.*myapp"          # match against full command line
pgrep -u alice                      # processes owned by alice
pgrep -P 1234                       # children of PID 1234
pkill gunicorn                      # SIGTERM all matching
pkill -9 gunicorn                   # SIGKILL all matching
pkill -HUP nginx                    # send a specific signal

Cleaner and safer than parsing ps. Use these.

top, htop, btop

top                                  # live process list
top -p $(pgrep -d, python)           # only python processes
htop                                 # friendlier; sortable, scrollable, mouse
btop                                 # newer, prettier

In top:

• 1 — show per-CPU breakdown
• M — sort by memory
• P — sort by CPU
• k — kill a process
• q — quit

htop adds tree view (F5), search (F3), and a column picker. Install it everywhere.

Foreground / background / jobs

long_running_command                 # foreground — terminal blocks until done
long_running_command &               # background — terminal returns immediately

jobs                                 # list jobs in this shell
jobs -l                              # also show PIDs
fg %1                                # bring job 1 to foreground
bg %1                                # resume job 1 in background
kill %1                              # kill job 1

A “job” is a shell concept (live in this shell), not a kernel concept. Closing the terminal kills jobs by default — see nohup and disown below.

Suspending and resuming

# In foreground:
Ctrl-Z                               # suspend (SIGTSTP) → stopped state
bg                                   # resume in background
fg                                   # resume in foreground

Ctrl-Z is “save my place, give me the prompt back.” Useful when you forgot to background something and don’t want to kill it.

nohup — survive logout

nohup ./long_script.sh &              # detach from terminal
# stdout → nohup.out
# immune to SIGHUP (sent when terminal closes)

Without nohup, closing the terminal sends SIGHUP to all child processes → they die. With nohup:

• SIGHUP is ignored.
• stdout/stderr redirected to nohup.out (or wherever you redirect).
• Process survives logout.

For long-running services, use systemd or a process supervisor — nohup is for one-off “I need this to keep running until I get back.”

disown — detach an already-running job

./script &
# realized you should logout
disown                                # detach from this shell — won't get SIGHUP
disown -h %1                          # don't send SIGHUP, but keep in jobs list

disown is nohup after the fact. Pairs with & for “keep this running after I logout.”

tmux / screen — terminal multiplexers

The robust way to keep things running:

tmux                                  # new session
tmux ls                               # list sessions
tmux attach -t name                   # reattach
tmux new -s mywork                    # named session

Inside tmux:

• Ctrl-b d — detach (session keeps running)
• Ctrl-b c — new window
• Ctrl-b n / p — next/prev window
• Ctrl-b " — split horizontally
• Ctrl-b % — split vertically

Reattaching gives you exactly what you left. Survives ssh disconnects, terminal closures, system reboots? No — server reboot kills tmux too. For real persistence, use systemd.

kill — send signals

kill 1234                            # SIGTERM (graceful) to PID 1234
kill -9 1234                         # SIGKILL (forced) to PID 1234
kill -HUP 1234                       # SIGHUP (often = reload config)
kill -USR1 1234                      # USR1 (app-defined, often = log rotation)
kill -l                              # list all signal names

killall gunicorn                     # kill all processes named gunicorn
killall -9 gunicorn                  # forced

Signals are deeply important — see 10_signals_traps.md.

kill -9 is the nuclear option. Try kill (TERM) first, give it a few seconds, then kill -9 if needed. SIGKILL skips cleanup (no chance for the app to flush buffers, close connections, persist state).

Finding what’s using a port / file

ss -tlnp                              # listening TCP sockets with PIDs
ss -anp | grep ':5432'                # all sockets touching port 5432
lsof -i :8000                         # processes using port 8000
lsof -p 1234                          # files opened by PID 1234
fuser /var/log/myapp.log              # who has this file open
fuser -k /var/log/myapp.log           # kill them (be careful)

ss (modern) replaces netstat. For interactive use both work; for scripts use ss (faster on busy hosts).

See ../28_networking/14_troubleshooting_tools.md.

Process tree

ps auxf                               # forest view (BSD)
ps -ejH                               # hierarchical
pstree -p                             # tree with PIDs
pstree -p alice                       # only alice's processes

Helpful when “this process won’t die” — you may be killing a child while the parent immediately respawns it.

Zombies and orphans

Many zombies = parent has a bug (not reaping). They’re harmless individually but eventually exhaust the PID table. Find them:

ps aux | awk '$8=="Z"'

Restart the parent to clean them up.

CPU and memory of one process

top -p 1234
ps -p 1234 -o pid,pcpu,pmem,rss,vsz,etime,cmd
pidstat -p 1234 1                     # 1-second snapshots, requires sysstat
htop -p 1234

For deep memory analysis: pmap 1234 shows memory map; smem (third-party) gives PSS (proportional set size — fairer than RSS for shared memory).

Limits

Per-process limits (ulimit):

ulimit -n                             # max open files (default often 1024)
ulimit -u                             # max processes
ulimit -v                             # max virtual memory
ulimit -n 65535                       # raise limit (only down without root)

Permanent limits in /etc/security/limits.conf. Many production “weird” issues come from default file-descriptor limits hitting workloads with thousands of connections.

Useful one-liners

# Top 10 processes by memory
ps aux --sort=-rss | head

# Total memory used by all python processes
ps aux | grep python | awk '{sum+=$6} END {print sum/1024 " MB"}'

# Kill anything matching a pattern (carefully)
pkill -f "old_script_name"

# Wait for a port to be open
until nc -z localhost 8000; do sleep 0.5; done; echo "ready"

# Count processes per user
ps -eo user | sort | uniq -c | sort -rn

# Parent of a process
ps -p $(pgrep gunicorn | head -1) -o ppid=

# Find runaway disk I/O
iotop                                  # requires root and `iotop` installed

Common pitfalls

• kill -9 for everything — skips cleanup, can corrupt state (DBs mid-write, file handles not flushed). Always TERM first.
• pkill -9 -f python — matches every Python process, including the one you’re running. Be specific.
• Killing a child while the parent respawns it — process supervisor (systemd, supervisord, foreman) is doing its job. Stop the parent or use systemd to manage the service.
• Creating tmux/screen sessions but not naming them — tmux ls shows 0: 1: 2: and you don’t remember which is which. Always tmux new -s name.

Common interview confusions

• “& and nohup do the same thing.” — & backgrounds in this shell; the job dies on SIGHUP (terminal close). nohup immunizes from SIGHUP. Use both for “keep running after I logout.”
• “kill -9 is the right way to kill a process.” — last resort. SIGKILL skips signal handlers; the app can’t clean up. Try TERM first, escalate to KILL only if needed.
• “top shows current CPU usage.” — by default, %CPU is averaged over the process’s lifetime, not the last second. Use top with the option for current.

Interview angle

• “How do you find what’s using port 8000?” — lsof -i :8000 or ss -tlnp | grep :8000. Either tells you the PID and command.
• “Difference between &, nohup, and disown?” — & backgrounds (still attached to terminal); nohup makes immune to SIGHUP (survives logout); disown detaches an already-running job from the shell. Combine & + nohup for “run in background, survive logout.”
• “How do you keep a script running across SSH disconnects?” — tmux or screen (interactive); nohup ... & (one-off); systemd unit (proper service).
• “Difference between kill and kill -9?” — default kill sends SIGTERM (process can clean up). -9 is SIGKILL (immediate, no cleanup). Always TERM first.
• “What’s a zombie process?” — child that exited but parent hasn’t reaped (called wait()). Holds a PID slot, no resources. Many zombies indicate a buggy parent — restart it.
• “How do you find the top processes by memory?” — ps aux --sort=-rss | head or htop (sort with F6 → MEM%). RSS is the meaningful column, not VSZ.