ML system design framework

The structure to apply to any “design an ML system” question. Having a framework matters more than knowing any particular design — it stops you diving into model architecture in minute two.

The order

1. Clarify the problem (5 min). Do not skip this.

• What decision does the prediction drive? If nobody acts on it, don’t build it.
• What does success look like in business terms?
• Scale: users, requests per second, corpus size, growth.
• Latency budget: batch, near-real-time, or synchronous?
• What data exists today, and what’s available at prediction time?

2. Frame it as an ML problem. Classification, regression, ranking, retrieval, generation? What’s the label, precisely? What’s the prediction horizon? See ../01_ml_foundations/01_ml_problem_types.md.

3. State the baseline. Rules, popularity, or a simple model. Everything is measured against it, and sometimes the baseline wins.

4. Metrics. One offline primary, guardrails, and the online business metric. Say how they connect. See ../04_model_evaluation/06_choosing_the_metric.md.

5. Data and features. Sources, volume, labels, and how you avoid leakage. Point-in-time correctness if there’s any temporal element.

6. Model. Start simple, justify complexity. This is where candidates want to start and should arrive fourth.

7. Serving. Batch vs online, latency budget, caching, fallback when the model is unavailable.

8. Evaluation and rollout. Offline eval, shadow, canary, A/B, guardrails, rollback.

9. Monitoring. Drift, outcome metrics, retraining trigger.

10. Failure modes. Cold start, feedback loops, adversaries, fairness.

You won’t cover all ten deeply. Say the outline first, then let the interviewer steer.

The capacity arithmetic

Do it out loud. It shapes the architecture.

100M users, 10 recommendation requests/day each
  = 1B requests/day ≈ 12k QPS average, ~40k peak

Scoring 1000 candidates per request at 12k QPS
  = 12M model evaluations/second   <- impossible for a deep model
  -> two-stage: cheap retrieval narrows to ~500, expensive ranking scores those

That calculation is the reason recommenders and search are two-stage. Deriving it beats reciting it.

The recurring shapes

Most ML system design questions are one of these:

Recognising the shape early gives you the skeleton. Retrieve-then-rank in particular covers search, recommendations, ads and RAG — see ../09_rag_embeddings/08_hybrid_search_and_reranking.md.

The two-stage pattern

millions of candidates
  -> RETRIEVAL: cheap, high recall, ~ms          (ANN, inverted index, heuristics)
  -> hundreds of candidates
  -> RANKING: expensive, high precision          (gradient boosting, cross-encoder, deep model)
  -> tens of results
  -> RE-RANK: business rules, diversity, freshness
  -> what the user sees

Each stage optimises a different metric — recall first, precision second — which dissolves the accuracy/latency trade rather than compromising on it. The final business-rules layer is where diversity, freshness and policy constraints live, and mentioning it signals product awareness.

Cold start

Asked in nearly every recommender or personalisation question.

The exploration point matters: a purely exploitative recommender never surfaces new items, so they never get interaction data, so they stay invisible. You need deliberate exploration budget.

Feedback loops

The failure mode specific to deployed ML: the model shapes the data it’s later trained on.

• A recommender trained on clicks it caused. Position bias in, position bias out.
• A fraud model that blocks transactions never learns whether they were fraudulent.
• Loan approvals only generate repayment data for approved applicants.

Mitigations: randomised exploration holdouts, propensity weighting, and holding back a small untreated population as a clean signal. Raising this unprompted is a strong senior signal.

Interview angle

• “How do you approach an ML system design question?” — clarify the decision and constraints, frame it as an ML problem with a precise label, state the baseline, define metrics, then data, model, serving, rollout, monitoring and failure modes. Say the outline first and let the interviewer pick the depth.
• “Why are recommenders and search two-stage?” — the capacity arithmetic. Scoring every candidate with an expensive model at production QPS is impossible, so a cheap high-recall stage narrows the field for an expensive high-precision one.
• “How do you handle cold start?” — different answers for new users (popularity, demographics, onboarding signals, bandits) and new items (content features plus deliberate exploration, because without exploration new items never accumulate the interactions they need).
• “What’s a feedback loop and why does it matter?” — the deployed model shapes its own future training data, so offline metrics improve while the system reinforces its existing biases. Mitigate with exploration holdouts, propensity weighting and an untreated control population.
• “Where do most candidates go wrong?” — starting at model architecture. The decision being driven, the label definition and what’s available at prediction time determine far more than the model class.