The ML lifecycle

The end-to-end shape of a project. Interviewers use this to check whether you’ve shipped something or only trained something — the gap between the two is where most ML value is lost.

The stages

problem framing -> data -> features -> train -> evaluate
      ^                                            |
      |                                            v
   monitor <- serve <- validate <- ------------ decide

It’s a loop, not a line. The unglamorous stages — framing, validation, monitoring — are where projects succeed or fail.

1. Problem framing

Covered in 01_ml_problem_types.md. The output of this stage is a written statement of: the decision being driven, the label definition, the prediction horizon, the metric, and the baseline to beat.

Define the baseline before you build anything. It’s usually one of:

• the current rule-based system,
• predicting the majority class or the mean,
• a simple logistic regression on three obvious features.

If your neural network beats a logistic regression by 1%, you’ve learned something important and it isn’t “ship the neural network”.

2. Data

The stage that consumes most of the time and gets least of the discussion.

• Availability at prediction time. For each candidate feature, ask when its value is known. If it’s populated after the event you’re predicting, it’s leakage.
• Volume and label quality. Sample a few hundred labels and check them by hand. Label noise puts a ceiling on achievable performance, and knowing where that ceiling is stops you chasing it.
• Class balance, which drives metric and sampling choices.
• Splitting strategy, decided now, not later — see 02_train_val_test_split.md.

3. Features

See ../03_feature_engineering/. The lifecycle-level concern is training/serving skew: the feature computed in your training notebook must be identical to the one computed in the serving path. Different code paths for the same feature is the most common cause of a model that performs worse in production than offline.

The structural fixes are a shared feature-computation library, or a feature store that serves both offline and online from one definition.

4. Train

Track every run — parameters, metrics, data version, code version — from the first experiment, not once things get serious.

import mlflow

with mlflow.start_run():
    mlflow.log_params({"max_depth": 6, "lr": 0.05})
    model.fit(X_train, y_train)
    mlflow.log_metric("val_ap", average_precision_score(y_val, model.predict_proba(X_val)[:, 1]))
    mlflow.sklearn.log_model(model, "model")

MLflow is the safe default answer here — it’s the most portable, works across sklearn/PyTorch/XGBoost/HuggingFace, and recent versions also handle prompt logging, evaluation and tracing for GenAI workloads. See ../15_mlops_llmops/.

Without tracking, “which run produced the model in production?” becomes unanswerable within about two weeks.

5. Evaluate

Against the metric you agreed in stage 1, plus:

• Segment breakdowns. An aggregate number hides a model that fails badly for one customer tier or region. This is both a quality issue and a fairness issue.
• Error analysis. Read fifty wrong predictions. It is the highest-information-per-hour activity in ML and it’s routinely skipped.
• Calibration, if you’re using the probabilities rather than just the ranking — see ../04_model_evaluation/.

6. Decide

Explicitly: does this beat the baseline by enough to justify the operational cost of running it? “Model is better” is not the same as “shipping is worth it”. A model needing a feature store, a serving cluster and an on-call rotation must clear a higher bar than one that writes a daily table.

7. Validate before serving

Shadow mode first: run the model on live traffic, log its predictions, act on none of them. It catches training/serving skew, latency problems and unexpected inputs with zero user risk.

Then a canary or A/B rollout, with a metric that reflects the business outcome rather than the model metric.

8. Serve

Choose the least operationally demanding option that satisfies the requirement. Batch is dramatically easier to run, debug and roll back.

9. Monitor

Models degrade silently. Nothing throws an exception when accuracy falls.

The gap between prediction and label arriving is the hard part. For fraud you might wait weeks for a chargeback; for churn, months. Until labels land, drift monitoring on the inputs and prediction distribution is your early-warning system.

• Data drift — the input distribution moved.
• Concept drift — the relationship between inputs and label moved. Harder, and only detectable once labels arrive.

Both require a retraining answer: scheduled cadence, or triggered by a drift threshold.

The LLM-application variant

The stages map over with different content:

Two things that are genuinely new: the provider can change the model under you, so pin versions and re-run evals on upgrade; and cost per request is variable and material, so token spend is a first-class production metric alongside latency.

Interview angle

• “Walk me through an ML project end to end.” — frame the decision and the baseline, get and inspect the data, decide the split, build features with serving in mind, train with experiment tracking, evaluate against the baseline with segment breakdowns and error analysis, shadow-deploy, roll out gradually, monitor inputs and outputs, plan retraining. Naming shadow mode and monitoring is what marks experience.
• “What’s the most common reason ML projects fail?” — not model quality. Poor problem framing, leakage, training/serving skew, or no path to acting on the prediction.
• “How do you know when to retrain?” — on a schedule matched to how fast the domain moves, or triggered by drift detection on input distributions and prediction distributions, with an outcome metric confirming it once labels arrive.
• “What is training/serving skew and how do you prevent it?” — the same feature computed differently in training and serving. Prevent it by sharing one implementation, or using a feature store that serves both paths from one definition. Shadow deployment catches what’s left.
• “Model looks great offline. What would stop you shipping it?” — no shadow validation yet, no monitoring in place, no rollback plan, operational cost exceeding the benefit over the baseline, or bad performance on a segment that matters.
• “How is the lifecycle different for an LLM feature?” — training is usually replaced by prompt and retrieval design; evaluation needs a curated eval set plus judge or human review; and you must pin the model version, because the provider can change it beneath you.