Latency Numbers and Performance Reference

The numbers that let you reason about a design without hand-waving. You don’t need them to the nanosecond — you need the order of magnitude and the ratios between them.

Latency numbers every engineer should know

Jeff Dean’s classic table, rounded to what matters:

The takeaways that drive design:

• Memory ≫ SSD ≫ disk ≫ network. Each step is ~10-100×.
• A network hop is ~500 µs minimum inside a datacenter — every microservice call, every DB query, every cache lookup pays this.
• Cross-region is ~100+ ms — physics. You can’t cache your way around the speed of light; you replicate data closer to users instead.
• Sequential ≫ random for both SSD and disk — why databases batch, why columnar formats win, why log-structured storage exists.

What this means for a design

A request’s latency budget

A user-facing request with a 200 ms p99 budget:

CDN/LB                ~5 ms
API server processing ~10 ms
1 cache lookup        ~1 ms   (network hop + Redis)
1 DB query (indexed)  ~5-10 ms
serialization         ~5 ms
─────────────────────────────
~30-40 ms with everything going right

That budget evaporates fast if you add hops. Every synchronous service call is ~1-10 ms minimum. A request that fans out to 5 services sequentially is 50 ms before any of them does real work — and the p99 is the slowest of the 5, not the average. This is why:

• You fan out in parallel, not sequentially.
• You cache to remove DB hops.
• You keep the synchronous call chain shallow (3+ deep is a smell).
• You push non-critical work async (queue) so it’s off the request path.

Throughput intuition

These aren’t precise — they’re “is this one box or a fleet?” anchors. If your capacity estimate says 500 write QPS and a Postgres primary does low-thousands, you don’t shard. If it says 50k write QPS, you do.

Availability numbers — the nines

Design implications:

• Three nines — a single well-run region with redundancy at each tier (multiple AZs, DB replica, stateless servers).
• Four nines — you’re now thinking about fast automated failover, no single points of failure, careful deploys.
• Five nines — multi-region active-active, near-zero-downtime everything; very expensive. Rarely actually required; don’t volunteer it.

In an interview, ask which SLO they want — it changes whether you need multi-region. Defaulting to five nines is over-engineering.

Combining components — availability math

Services in series multiply: a request through API (99.9%) → cache (99.9%) → DB (99.9%) has ~99.7% availability (0.999³). Every dependency you add on the critical path lowers availability.

Redundancy in parallel raises it: two independent DB replicas each at 99% give ~99.99% combined (1 − 0.01²) — if failures are independent and failover works.

The lessons: shorten the critical-path dependency chain; add redundancy at the tiers that matter; and remember that “failover works” is an assumption you must actually test.

Consistency vs latency — the unavoidable trade

You cannot have strong consistency and the lowest possible latency and availability during a partition. (CAP, and more practically PACELC: even without a partition, you trade latency for consistency.)

• Strong consistency — reads see the latest write. Costs a coordination round-trip (quorum, leader read). Use where correctness demands it (balances, inventory).
• Eventual consistency — reads may be stale for a window. Cheaper, faster, more available. Use where staleness is tolerable (feeds, counts, search indexes).

In a design, say per-data-type where you land: “the account balance is strongly consistent; the activity feed is eventually consistent with a read-your-writes exception for the author.”

Percentiles — design for p99, not the average

Averages lie. A service with a 10 ms average can have a 500 ms p99 — and at scale, p99 is the experience of millions of requests. Worse: in a fan-out request, the p99 of the overall request approaches the p99 of the slowest dependency, because you only need one slow leg.

• Quote and design against p99 / p99.9, not the mean.
• Tail-tolerant patterns — hedged requests (fire a backup after a delay), timeouts + fallbacks, request coalescing.
• A slow tail is often a queue somewhere — connection pool exhaustion, GC pause, a hot shard.

Interview angle

• “How fast is a network call?” — ~500 µs round trip inside a datacenter, ~100+ ms cross-region. Every cache lookup, DB query, and service call pays at least the datacenter hop. It’s why you fan out in parallel and keep call chains shallow.
• “Memory vs SSD vs disk vs network — order of magnitude?” — memory ~100 ns, SSD ~10s of µs, disk seek ~10 ms, datacenter network ~500 µs, cross-region ~100+ ms. Each tier is roughly 10-100× the one before. This ordering is why caching works and why sequential I/O beats random.
• “What does 99.99% availability require?” — ~52 min downtime/year; no single points of failure, multiple AZs, automated fast failover, careful deploys. Five nines pushes you to multi-region active-active — expensive, rarely actually needed. Ask which SLO they want before designing for it.
• “Why design against p99 instead of average?” — at scale the tail is millions of real requests, and in a fan-out request the overall p99 tracks the slowest dependency, not the average. Averages hide the queueing and contention that actually hurt users.
• “Your design has 4 services on the request path, each 99.9%. What’s the availability?” — they multiply: ~0.999⁴ ≈ 99.6%. Every critical-path dependency lowers availability — shorten the chain, or add redundancy at the tiers that matter.