grep, sed, awk

The classic Unix text-processing trio. grep finds lines matching patterns, sed does line-by-line stream edits, awk is a small language for column/field processing. Knowing the common idioms saves writing 50-line Python scripts.

grep — find matching lines

grep "pattern" file
grep -i "pattern" file              # case-insensitive
grep -v "pattern" file              # invert: lines NOT matching
grep -n "pattern" file              # show line numbers
grep -c "pattern" file              # count matching lines
grep -l "pattern" *.log             # list filenames with matches
grep -L "pattern" *.log             # list filenames WITHOUT matches
grep -r "pattern" /path             # recursive
grep -E "pat1|pat2" file            # extended regex (-E or use egrep)
grep -F "literal string" file       # fixed string, no regex (faster, safe with $)
grep -o "pattern" file              # print only the match, not the whole line
grep -A 3 "pattern" file            # 3 lines after match
grep -B 3 "pattern" file            # 3 lines before
grep -C 3 "pattern" file            # 3 lines around

Useful grep patterns

# Match lines starting with "ERROR"
grep "^ERROR" log.txt

# Lines ending with "OK"
grep "OK$" log.txt

# Empty lines
grep -E "^$" file

# Lines with at least one digit
grep "[0-9]" file

# Word boundary
grep -w "log" file                  # matches "log", not "blog"

# Multiple patterns (alternation)
grep -E "error|warning|fatal" log.txt

# Show context around a match
grep -n -C 5 "exception" log.txt

ripgrep — modern grep

rg is faster, respects .gitignore by default, has better defaults:

rg "TODO"                            # search current dir, recursive, ignores .git/.venv/node_modules
rg "TODO" --type py                  # only .py files
rg -i "error"                        # case-insensitive
rg "pattern" --no-ignore             # search ignored files too
rg --files | grep something          # list files instead of matching

For interactive use, rg is almost always the right answer. grep is universal in scripts.

sed — stream editor

sed 's/old/new/' file               # replace first 'old' on each line
sed 's/old/new/g' file              # replace all on each line
sed -i 's/old/new/g' file           # in-place edit (Linux)
sed -i '' 's/old/new/g' file        # in-place edit (macOS — needs '')
sed -E 's/[0-9]+/N/g' file          # extended regex
sed -n '10,20p' file                # print lines 10-20 only
sed '5d' file                       # delete line 5
sed '/^$/d' file                    # delete empty lines
sed '/pattern/d' file               # delete lines matching pattern

sed substitution syntax

s/PATTERN/REPLACEMENT/FLAGS

Use any delimiter (avoids escaping / in paths):

sed 's|/usr/bin|/usr/local/bin|g' file
sed 's#http://#https://#g' file

Backreferences (\1, \2):

echo "foo bar baz" | sed -E 's/(\w+) (\w+)/\2 \1/'    # bar foo baz
sed -E 's/^(.+)\.bak$/\1/' files                       # strip .bak suffix

sed gotchas

• macOS sed is BSD, Linux is GNU — -i syntax differs (sed -i '' on macOS, sed -i on Linux).
• No \d, \w, \s in basic sed — use [0-9], [a-zA-Z_], [[:space:]].
• -E for extended regex (otherwise (, ), {, }, + need backslashes).
• -i rewrites the file unconditionally, even on errors. Run without -i first to preview.

awk — field processing

awk reads lines, splits each into fields (default by whitespace), runs your code per line:

awk '{print $1}' file               # first field of each line
awk '{print $1, $3}' file           # fields 1 and 3 (comma → output separator)
awk '{print NF, $0}' file           # NF = number of fields, $0 = whole line
awk 'NR==5 {print}' file            # print only line 5 (NR = current line number)
awk 'END {print NR}' file           # total line count
awk '/error/ {print}' file          # only lines matching pattern
awk 'NF > 0' file                    # non-empty lines
awk -F: '{print $1}' /etc/passwd    # custom field separator (colon)
awk '{sum += $1} END {print sum}' numbers.txt

awk variables

Common awk idioms

# Sum a column
awk '{s+=$1} END {print s}' data.txt

# Average
awk '{s+=$1; n++} END {print s/n}' data.txt

# Max of column
awk 'NR==1 || $1 > max {max=$1} END {print max}' data.txt

# Print unique values in column 1
awk '!seen[$1]++' data.txt

# Print column 2 where column 1 == "ERROR"
awk '$1 == "ERROR" {print $2}' log.txt

# Filter on multiple fields
awk '$3 > 100 && $4 == "active"' data.txt

# Substitute and print
awk '{gsub(/old/, "new", $2); print}' data.txt

# CSV (comma-separated)
awk -F, '{print $2}' data.csv

# Print with custom OFS
awk -F: -v OFS=, '{print $1, $3, $7}' /etc/passwd

A real example: top IPs from access log

awk '{print $1}' access.log | sort | uniq -c | sort -rn | head

Pipeline:

1. awk '{print $1}' — first field (IP address).
2. sort — group same IPs together.
3. uniq -c — count consecutive duplicates.
4. sort -rn — sort by count, descending.
5. head — top 10.

This 1-liner replaces a 20-line Python script. Memorize it.

When awk beats Python

For simple column-based work on big files, awk is:

• Faster (compiled regex, low overhead).
• Already installed.
• One line.

For anything beyond ~5 lines of awk, switch to Python — readability collapses fast.

Common combinations

# Find error lines, extract a field, count
grep -i error log.txt | awk '{print $5}' | sort | uniq -c | sort -rn

# Replace all occurrences of foo with bar in *.py
find . -name "*.py" -exec sed -i 's/foo/bar/g' {} +

# Show only non-comment, non-blank lines of a config
grep -v '^#' /etc/nginx/nginx.conf | grep -v '^$'

# Show the 10 largest files in a tree
find . -type f -printf "%s\t%p\n" | sort -rn | head -10

Common pitfalls

• sed -i without backup wipes the original. For risky edits, sed -i.bak keeps file.bak.
• Forgetting g flag in sed s/// — only the first match per line is replaced.
• grep regex vs glob — grep "*.txt" doesn’t do what you think. grep "\.txt$" matches lines ending in “.txt”.
• awk’s automatic field splitting on whitespace — multiple spaces are treated as one. Use -F to set the separator if your data uses commas/tabs.
• grep "pattern with space" without quotes — shell word-splits. Always quote.

Common interview confusions

• “grep, sed, awk are interchangeable.” — overlap, but: grep is for finding lines, sed is for line-by-line transforms, awk is a small language for field-oriented processing.
• “sed substitutes all occurrences by default.” — first per line. Add g flag.
• “awk needs a Python-style for loop.” — awk auto-loops over input lines. The body of '{...}' runs once per line.

Interview angle

• “How would you find lines containing ‘error’ in a log, with the 5 lines after each match?” — grep -A 5 -i error log.txt.
• “Replace all ‘foo’ with ‘bar’ in every .py file recursively.” — find . -name "*.py" -exec sed -i 's/foo/bar/g' {} +. Test without -i first.
• “Top 10 IPs in an access log?” — awk '{print $1}' access.log | sort | uniq -c | sort -rn | head. The classic pipeline.
• “Difference between basic and extended regex in grep / sed?” — basic requires backslashes for +, ?, (, ), {, }. Extended (-E or egrep) treats them as metacharacters by default.
• “When would you reach for awk over Python?” — simple column work on streams or huge files. For more than ~5 lines of awk, switch to Python.
• “How do you delete lines matching a pattern in-place?” — sed -i '/pattern/d' file. Linux syntax; macOS needs sed -i '' '/pattern/d' file.
• “What’s awk '!seen[$1]++' doing?” — prints first occurrence of each value of column 1 (deduplicates by column 1, preserving order). Common idiom.