Network Troubleshooting Tools

The tools that come up in “your service can’t reach the database, what do you do?” questions. Knowing which to reach for in which order is the actual skill.

Top-down debugging order

When “A can’t reach B,” check from the highest layer down:

1. DNS — does B resolve at all? dig, nslookup. See 05_dns.md.
2. L4 reachability — can you open a TCP connection to B:port? nc, telnet, curl.
3. L3 reachability — can you reach the host at all? ping, traceroute.
4. L7 — does the app respond correctly once connected? curl -v, httpie, app logs.
5. Captures — packet-level: tcpdump, tshark, wireshark.

DNS: dig

dig api.example.com                  # default A record
dig api.example.com AAAA             # IPv6
dig api.example.com MX               # mail records
dig +short api.example.com           # just the answer
dig api.example.com @8.8.8.8         # ask Google's resolver instead
dig +trace api.example.com           # full root → TLD → authoritative chain
dig +tcp api.example.com             # over TCP (test if UDP/TCP differs)

Useful when “DNS works on my laptop but not on the server” — dig @<server-resolver> to see what that resolver returns.

L3 reachability: ping, traceroute/tracert, mtr

ping example.com                  # ICMP echo, RTT
ping -c 5 example.com             # 5 packets and stop
traceroute example.com            # show every hop
mtr example.com                   # combined ping+traceroute, continuous

mtr is the best of the three for production debugging — shows you packet loss per hop. If hop 5 has 50% loss, you’ve localized the problem to that router/network segment.

ICMP is often blocked (cloud security groups, corporate firewalls) — a failing ping doesn’t mean the host is down, it might mean ICMP is filtered. Always confirm with a TCP-level check too.

L4 reachability: nc, telnet, curl

nc -vz example.com 443            # can I open TCP to port 443?
nc -vz -u 8.8.8.8 53              # UDP equivalent
telnet example.com 443            # older but works
curl -v telnet://example.com:443  # curl as a TCP probe
curl -v https://example.com/      # full HTTP+TLS test

If nc -vz host port succeeds but your app can’t connect, the problem is in the app/library (cert validation, timeout, auth), not the network.

For TLS specifically:

openssl s_client -connect example.com:443 -servername example.com   # show cert chain, ciphers
curl -vI https://example.com/                                       # show TLS + headers

-servername matters: without SNI (12_tls_https_certificates.md), the server may serve the wrong cert.

What’s listening locally: ss, netstat, lsof

ss -tlnp                          # TCP, listening, numeric, processes
ss -anp | grep 5432               # all sockets touching port 5432
ss -s                             # summary: how many ESTABLISHED, TIME_WAIT, etc.
netstat -tlnp                     # older equivalent (deprecated on modern Linux)
lsof -i :8000                     # which process owns port 8000
lsof -i -P -n | grep LISTEN       # all listeners with process names

ss replaced netstat and is faster on modern kernels. Use ss first.

Common diagnostics:

• “Why isn’t my app reachable?” — ss -tlnp | grep <port>. If empty, the app didn’t bind. If bound to 127.0.0.1:8000 instead of 0.0.0.0:8000, only local connections work.
• “Are we leaking connections?” — ss -s or ss -ant | wc -l. Steadily growing = leak.
• “Lots of TIME_WAIT?” — short-connection workload, missing keep-alive. See 02_tcp_vs_udp.md.

HTTP: curl, httpie

curl -v https://api.example.com/             # verbose: handshake, headers, body
curl -I https://api.example.com/             # HEAD only — just response headers
curl -w "@curl-format.txt" -s -o /dev/null https://api.example.com/   # timing breakdown
curl --resolve api.example.com:443:1.2.3.4 https://api.example.com/   # bypass DNS
http GET api.example.com/users               # httpie — colorized JSON

curl-format.txt:

time_namelookup:    %{time_namelookup}\n
time_connect:       %{time_connect}\n
time_appconnect:    %{time_appconnect}\n
time_pretransfer:   %{time_pretransfer}\n
time_starttransfer: %{time_starttransfer}\n
time_total:         %{time_total}\n

Tells you which phase is slow — DNS? TCP connect? TLS? First byte? Total?

--resolve is a lifesaver when DNS isn’t yet pointing at the new endpoint and you want to test it: forces curl to use a specific IP for a hostname.

Packet capture: tcpdump, tshark, Wireshark

tcpdump -i eth0 -nn port 5432                       # all Postgres traffic
tcpdump -i any -nn host 10.0.0.5 and port 443       # to/from a specific host on TLS
tcpdump -i any -nn -w capture.pcap -c 1000          # save to file (open in Wireshark)
tcpdump -i any -nn -A port 80                       # ASCII payload (HTTP plaintext)

When higher-level tools say “connection reset” but you don’t know who reset it, only tcpdump will tell you — you’ll see the RST flag and which side sent it.

Cloud-specific: VPC Flow Logs

In AWS, every accepted/rejected packet flow can be logged to CloudWatch / S3:

2 123456789012 eni-abc 10.0.1.5 10.0.20.5 53924 5432 6 5 1234 ... ACCEPT OK

Schema fields: VPC, ENI, source IP, dest IP, source port, dest port, protocol, packets, bytes, action (ACCEPT/REJECT). Filter on REJECT to find blocked flows.

GCP has VPC Flow Logs with similar semantics. Azure has NSG Flow Logs.

Routing inspection

ip route                               # routing table
ip addr                                # interfaces and IPs
ip neigh                               # ARP table (who's at this MAC?)
route -n                               # older equivalent

If the route table for 10.0.0.0/8 points to the wrong gateway, packets vanish. traceroute shows where.

Bandwidth: iperf3, iftop, nload

# server side
iperf3 -s

# client side
iperf3 -c server.example.com -t 30

Tests raw TCP throughput between two hosts. Useful when “the network feels slow” — does the link itself give you 1 Gbps or 100 Mbps?

iftop shows live per-flow bandwidth on an interface. nload shows aggregate.

DNS-specific debugging

dig +trace api.example.com         # follow the chain from root down
dig +nssearch example.com          # find authoritative nameservers
host api.example.com               # one-line answer
nslookup api.example.com 8.8.8.8   # query a specific resolver
systemd-resolve --status           # what resolvers does this host use? (systemd)
cat /etc/resolv.conf               # the simpler version

In Kubernetes, DNS issues are common. The pod’s /etc/resolv.conf points at the cluster CoreDNS at 10.x.0.10. Bugs to check:

• search lines accidentally resolving short names to wrong namespaces.
• CoreDNS pod down.
• Search domain confusion.

Process / connection tracing

strace -e trace=network -p <pid>            # trace network syscalls of a running process
ltrace -p <pid>                             # trace library calls
tcpflow -i any port 5432                    # reassemble TCP streams to plain text

strace shows you exactly which syscall (and on which fd) is hanging — sometimes the answer is “the app called connect() and the kernel is waiting on TCP retransmit, your network is dropping packets.”

When to reach for what — quick map

Interview angle

• “Walk me through debugging ‘service A can’t reach service B’.” — top-down: DNS resolves? → TCP connects (nc -vz)? → ICMP / mtr? → app-level (curl -v with full headers)? → packet capture if needed. Mention checking firewall/SG and the listening process on B.
• “netstat vs ss?” — ss is the modern replacement, faster on big socket tables. Same kind of output.
• “ping works but my app can’t connect — what’s possible?” — ICMP is open but TCP to that port is not (firewall), or app isn’t listening, or it’s listening on 127.0.0.1 only. Check ss -tlnp on the target and the firewall rules.
• “TCP connection times out — what tools tell you why?” — mtr to see where packets are lost en route; tcpdump to see if SYNs go out and any responses arrive; cloud Flow Logs to see if a SG/NACL is silently dropping.
• “How do you debug a TLS handshake failure from the command line?” — openssl s_client -connect host:443 -servername host shows cert chain, cipher negotiated, and any SNI/cert mismatch errors.
• “What’s the value of VPC Flow Logs?” — they show ACCEPT/REJECT per flow, so you can see which layer dropped a packet (SG vs NACL vs no route) — invisible to in-instance tools.