Replication, sharding and consistency models

The scaling vocabulary. Most system design answers need these words used precisely, and imprecision here is very visible.

Replication

Copies of the same data on multiple nodes, for read scaling and fault tolerance.

Single-leader is the default and what Postgres and MySQL do out of the box. Multi-leader appears in multi-region deployments and brings write conflicts that need resolution — last-write-wins loses data silently, so CRDTs or application-level merge are the honest answers.

Synchronous or asynchronous

The choice that decides your failure behaviour:

Semi-synchronous is the common compromise: wait for one replica, let the rest catch up asynchronously. Bounded data loss, bounded latency.

Replication lag — the bug that reaches users

Asynchronous replicas are behind. That produces a specific, very common bug: a user updates their profile, the write goes to the leader, the subsequent read goes to a replica that hasn’t caught up, and the UI shows the old value.

Fixes, in order of preference:

• Read-your-writes: route a user’s reads to the leader for a few seconds after they write. Cheap and effective.
• Monotonic reads: pin a user to one replica so they never see time move backwards.
• Wait for the replica to reach the write’s LSN before reading.

Naming read-your-writes unprompted is a good signal — it’s the failure people actually hit.

Sharding

Splitting data across nodes so each holds a subset. Replication copies; sharding divides.

The shard key is the decision that’s hardest to reverse. Choose it so the common query includes it — otherwise every read becomes a scatter-gather across all shards, which is slower than not sharding.

The classic failure: shard users by user_id, then need “all orders in the last hour across all users”. That query touches every shard.

What sharding costs:

• Cross-shard joins are gone or expensive.
• Cross-shard transactions need two-phase commit or a saga.
• Rebalancing moves data while serving traffic.
• Hotspots — one celebrity user can overwhelm a shard regardless of how even the hash is.

Shard last. Read replicas, caching, a bigger box, and archiving cold data all come first. Modern hardware and Postgres go a very long way, and saying so is better than reflexively sharding a hypothetical system.

CAP, and what it actually says

Under a network partition, choose consistency or availability.

The precision that matters: CAP is about behaviour during a partition, not a general “pick two”. With no partition you get both. And “consistency” in CAP means linearizability, which is stricter than the C in ACID.

PACELC is the more useful framing: during a Partition, choose Availability or Consistency; Else (normally), choose Latency or Consistency. That second half describes the trade-off you make every day, whereas partitions are rare.

The consistency spectrum

Ordered strongest to weakest:

Eventual consistency is a much weaker promise than people assume — it says nothing about when, and permits reading arbitrarily stale data meanwhile.

Causal consistency is the sweet spot for many systems: it prevents the confusing anomalies (seeing a reply before the message it replies to) without linearizability’s coordination cost.

Quorums

W + R > N   guarantees a read overlaps a write

With N=3: W=2, R=2 is the balanced choice. W=3, R=1 optimises reads at the cost of write availability. W=1, R=1 is fast and gives no consistency guarantee.

The caveat worth knowing: quorum overlap alone isn’t linearizability. Concurrent writes, failed writes that partially applied, and read-repair timing all leave edge cases. Dynamo-style systems are eventually consistent even with strict quorums.

Interview angle

• “Replication or sharding?” — different problems. Replication copies data for read scaling and fault tolerance; sharding divides it for write and storage scaling. Most systems need replication long before sharding.
• “A user updates their profile and immediately sees stale data. Why?” — replication lag: the write went to the leader, the read hit a lagging replica. Fix with read-your-writes consistency, routing that user’s reads to the leader briefly after a write.
• “How do you choose a shard key?” — so the common query contains it. If typical reads don’t include the shard key, every query becomes a scatter-gather across all shards and you’ve made things slower. It’s the hardest decision to reverse.
• “Explain CAP.” — under a network partition you must choose availability or consistency. It’s not a general “pick two” — with no partition you get both. PACELC is more useful: during a partition choose A or C, else choose latency or consistency.
• “What does eventual consistency actually promise?” — that replicas converge given no further writes. It says nothing about when, and permits arbitrarily stale reads meanwhile. Causal consistency is usually the better target: it rules out the confusing anomalies without full coordination cost.
• “When would you shard?” — after read replicas, caching, vertical scaling and archiving are exhausted. Sharding costs you cross-shard joins, distributed transactions, rebalancing complexity and hotspot risk. Modern single-node Postgres handles more than people expect.