DNS

DNS turns names like api.example.com into IPs like 203.0.113.42. It’s the first thing every HTTP request does, and it fails in surprising ways more often than people expect.

The lookup, end to end

Python app: requests.get("https://api.example.com/...")
   ↓
1. Check local cache (process / OS / glibc nscd)
2. Ask configured resolver  (e.g. 8.8.8.8 or 1.1.1.1)
3. Resolver asks root servers (.) — "who handles .com?"
4. Resolver asks .com TLD     — "who handles example.com?"
5. Resolver asks example.com authoritative — "what's api.example.com?"
6. Resolver returns the IP, caches it for the TTL
   ↓
Python now has 203.0.113.42 and opens TCP to it

Most of the time you only see step 1 (the cache hit). Cold caches do all of steps 1–6 — adds 20–200ms before your first packet flies.

Recursive vs authoritative resolvers

Your laptop talks to a recursive resolver. The resolver talks to a chain of authoritative servers.

Record types you’ll actually see

ALIAS / ANAME are vendor extensions (Route 53, Cloudflare) that act like CNAME but work at the zone apex.

TTL — the source of every “DNS hasn’t propagated yet” complaint

Every record has a Time To Live in seconds. Resolvers cache the answer for that long. When you change a record, clients keep using the old value until their cache expires — and you can’t force them to flush.

Practical TTL choices:

Pre-migration ritual: lower TTL to 60s a day before the cutover. Make the change. Wait 60s. Old TTL is in flight in caches up to its previous value, so plan for the higher of the two.

CNAME pitfalls

• CNAME at zone apex is illegal in classic DNS. You can’t example.com → other.com via CNAME. Use ALIAS (Route 53) or just A records.
• CNAME chains are slow — every chain step is a separate query. Keep them short.
• CNAME can’t coexist with other records on the same name. example.com IN CNAME and example.com IN MX is invalid.

Caching layers (where DNS hides)

When you change a record, the answer can be cached at:

1. The application’s resolver (Python socket.getaddrinfo doesn’t cache, but libraries on top might).
2. The OS resolver (nscd, systemd-resolved, macOS DirectoryService).
3. The recursive resolver (your ISP / 1.1.1.1).
4. Intermediate caching resolvers (corporate networks).

Each layer respects its own TTL. “Why is half the world seeing the old IP?” — because some resolver in the chain is still inside its TTL.

Python DNS gotchas

import socket
socket.gethostbyname("api.example.com")   # one IP, no caching, no async
socket.getaddrinfo("api.example.com", 443, type=socket.SOCK_STREAM)  # all IPs + records

• getaddrinfo is blocking. In an async app, wrap with loop.getaddrinfo() or use aiodns.
• Python doesn’t cache resolutions. requests/httpx/aiohttp rely on the OS resolver each time.
• /etc/hosts overrides everything. Useful for local testing, surprising in containers.
• The DNS record may have multiple IPs (load balancers often round-robin). Most clients pick the first; some retry on failure to others.

DNS and HA — round-robin DNS vs anycast

Two ways to give one name multiple IPs:

Round-robin is fine for “two web servers behind one name.” For real failover, you need health-checked DNS (Route 53 with health checks) or, better, a load balancer in front of the IPs.

Common production DNS failures

• TTL of an hour, you changed the record, half users see the old value for ~1h. No fix; wait it out.
• Resolver returns NXDOMAIN, app caches negative result for 60s+ (Linux negative-time-to-live). New record won’t be picked up immediately.
• search domain in /etc/resolv.conf turns redis into redis.svc.cluster.local then redis.example.com etc. — accidental wrong-namespace lookups in Kubernetes.
• DNS over TCP fallback is blocked by the firewall. Most queries fit in a UDP packet (<512 bytes), but DNSSEC and large records overflow and need TCP.

Useful tools

dig api.example.com              # full record details
dig +short api.example.com       # just the answer
dig api.example.com @8.8.8.8     # ask a specific resolver
dig +trace api.example.com       # show every step (root → TLD → authoritative)
nslookup api.example.com         # older equivalent
host api.example.com             # one-line answer

For continuous lookup: watch -n 1 dig +short .... See 14_troubleshooting_tools.md.

Interview angle

• “What happens when you type example.com into your browser?” (the classic) — DNS lookup (cache → recursive resolver → root → TLD → authoritative) → TCP handshake → TLS → HTTP.
• “Difference between A and CNAME?” — A is a name → IP, CNAME is name → name (alias). CNAME chain costs extra lookups; can’t be at zone apex.
• “What’s TTL and what does it cost you?” — how long resolvers cache the answer. Low TTL = faster failover, higher query load on authoritative. High TTL = cheaper, slower to react.
• “You changed an A record an hour ago and some users still see the old IP — why?” — they’re inside the previous TTL window of some resolver in the chain. Lowering TTL after the change doesn’t help; it had to be lowered before.
• “How would you do DNS-based failover?” — Route 53 (or similar) with health checks; lower TTL (60s); accept the failover delay equals the TTL plus client cache lag. Better: put a load balancer in front and skip DNS-level failover.