Design: a fraud detection system

The canonical classical-ML system design. It exercises imbalance, cost-sensitive thresholds, real-time features, label delay and adversarial drift — most of the hard parts in one problem.

Clarify

• Decision: block, allow, or send to manual review. Three outcomes, not two.
• Scale: 10k transactions/second peak.
• Latency: synchronous, inside the payment flow — under 100ms.
• Base rate: ~0.1% fraudulent.
• Costs: a missed fraud averages $500; a false positive costs a review (~$5) and customer friction.
• Labels: chargebacks arrive 30-90 days later. Manual reviews give faster partial labels.

The cost asymmetry (100:1) and the label delay are the two facts that drive everything downstream.

Frame it

Binary classification producing a score, not a decision. The score maps to three actions by two thresholds:

score < 0.3          -> allow
0.3 <= score < 0.85  -> manual review queue
score >= 0.85        -> block

Thresholds come from the cost matrix and the review team’s capacity, not from F1. If the team can review 2,000 cases/day, that caps the middle band regardless of what the model says. See ../04_model_evaluation/02_precision_recall_f1.md.

Metric: PR-AUC, not ROC-AUC. At 0.1% positives the true-negative pool is enormous, so a large increase in false positives barely moves FPR while precision collapses. See ../04_model_evaluation/03_roc_auc_vs_pr_auc.md.

Architecture

transaction -> feature service ──> model ──> score ──> decision
                    │                                     │
        ┌───────────┴────────────┐                        v
   online store            streaming aggregates      allow / review / block
   (Redis: profile,        (Flink: velocity                    │
    card, merchant)         counters, last 5m)                 v
                                                        review outcomes
                                                        (fast labels)

Features

Grouped by how they’re computed, because that determines the infrastructure:

Velocity features are the most predictive and the hardest to serve. “Five transactions in the last two minutes” needs a streaming pipeline (Kafka plus Flink) writing counters to the online store — you cannot compute it from a warehouse at request time.

Point-in-time correctness is essential. Training rows must use feature values as of that transaction’s timestamp. Joining current profile state onto historical transactions leaks the outcome backwards. This is where a feature store earns its cost. See ../03_feature_engineering/06_feature_stores.md.

Model

Gradient boosting — LightGBM or XGBoost. It wins on tabular data, trains fast enough for frequent retraining, handles missing values natively, and inference is well under the latency budget. See ../02_classical_ml/05_gradient_boosting.md.

Handle imbalance with scale_pos_weight plus threshold tuning, not SMOTE. Note that re-weighting distorts probabilities, so recalibrate if you use the score in an expected-loss calculation. See ../03_feature_engineering/05_imbalanced_data.md.

Keep a rules layer alongside the model. Rules encode known patterns instantly, are explainable to compliance, and cover the model’s blind spots on brand-new attack patterns:

if velocity_5m > 10 or amount > user.p99_amount * 10:
    return BLOCK                       # rules fire first, deterministic
return model_decision(score)

Rules-plus-model is the production reality, and proposing only a model is a gap.

Latency

100ms budget, roughly:

feature fetch (online store, parallel)   ~20ms
derived features                          ~5ms
model inference                          ~10ms
rules + decision                          ~5ms
logging (async, off the path)              0ms
                                     ---------
                                        ~40ms   comfortable

Fetch features in parallel, not sequentially. Have a fallback: if the feature store times out, score on transaction-only features and lean on rules. A payment flow cannot hang because the model is unavailable — degraded is better than down.

The label delay problem

Chargebacks take 30-90 days. Consequences:

• You cannot evaluate today’s model on today’s data. Accept a 30-90 day lag on the true metric.
• Manual review outcomes are fast partial labels — biased toward the reviewed band, but timely. Weight accordingly.
• Monitor proxies meanwhile: score distribution, feature drift, block rate, review-queue depth. See ../15_mlops_llmops/02_monitoring_and_drift.md.

Adversarial drift

Unlike most ML problems, the data-generating process is an adversary who adapts to your model.

• Retrain frequently — weekly or faster.
• Expect concept drift by default, not as an anomaly.
• Don’t expose scores or reasons to users; that’s a feedback channel for attackers.
• Keep a rules layer for rapid response — a new attack pattern can be blocked in an hour, retraining takes days.
• Watch for blocked-transaction blind spots: you never learn whether blocked transactions were actually fraud. Consider letting a tiny random sample through to maintain an unbiased signal, if the business tolerates it.

That last point is a genuine feedback-loop problem and a strong thing to raise.

Interview angle

• “Design a fraud detection system.” — score-not-decide with two thresholds mapping to allow/review/block, gradient boosting on transaction, profile, velocity and graph features, streaming aggregates for velocity, sub-100ms serving with a degraded fallback, and a rules layer alongside the model.
• “Why PR-AUC rather than ROC-AUC?” — at 0.1% positives the true-negative pool is huge, so false positives barely move FPR while precision collapses. PR-AUC has no true-negative term and reflects what reviewers experience.
• “Where do the thresholds come from?” — the cost matrix and review capacity. A 100:1 cost ratio and a team that can review 2,000 cases a day determine the bands, not a symmetric metric.
• “Which features are hardest to serve, and why?” — velocity features. “Five transactions in two minutes” requires a streaming pipeline writing to an online store; you can’t compute it from the warehouse inside a 100ms budget.
• “Labels arrive 60 days late. How do you know the model still works?” — you don’t, directly. Use manual review outcomes as faster biased labels, and monitor proxies: score distribution, feature drift, block rate, queue depth. Accept the lag on the true metric.
• “What’s different about fraud versus other ML problems?” — the data-generating process is adversarial and adapts to you. Concept drift is the norm, retraining is frequent, a rules layer gives you hour-scale response, and blocking creates a blind spot where you never learn the counterfactual.