Online evaluation and experiments

Offline eval tells you whether a change looks better on your test set. Online eval tells you whether it helped. They disagree more often than people expect.

Why offline and online diverge

The honest framing: offline metrics are a proxy, and the proxy is only good if a change in it predicts a change in the business metric. Validate that link once by comparing a few offline deltas against their A/B outcomes, then trust it accordingly.

Signals available in production

Implicit signals beat explicit feedback. Under 1% of users click thumbs-down, and those who do skew angry. A rephrase-and-retry rate is higher-volume and more honest.

Log the negatives specifically: escalations, retries and abandonments are your pipeline of future eval cases. See 02_building_eval_sets.md.

Shadow deployment

Run the new version on live traffic, log its output, serve the old version’s. Zero user risk.

It catches what offline eval cannot: real query distribution, real latency under load, unexpected inputs, and cost at real volume. For anything with side effects it’s the only safe way to validate before switching.

response = current.generate(query)
asyncio.create_task(log_shadow(candidate.generate(query), query))  # fire and forget, bounded
return response

Compare the two offline, ideally with a pairwise judge on the same inputs — a naturally paired comparison, which has far more statistical power than comparing two independent samples.

A/B testing LLM features

The standard machinery applies, with a few wrinkles specific to this domain.

Randomise on the user, not the request. A user seeing two different assistants across turns has an incoherent experience and pollutes the measurement.

Expect high variance. LLM output quality varies a lot per request, so you need more traffic than for a button-colour test to detect the same relative effect.

Guardrail metrics are mandatory. A quality win that doubles cost or adds three seconds of latency is usually not a win.

Primary:    task completion rate
Guardrails: p95 latency < 4s
            cost per conversation < $0.05
            escalation rate not worse
            refusal rate within ±2pp

Watch novelty effects. A new capability gets tried because it’s new. Run long enough for that to settle before believing the number.

The peeking, multiple-comparison and effect-size cautions are the ordinary ones — see ../00_math_foundations/02_probability_statistics.md.

Rolling out safely

1. Shadow — no user impact, compare offline.
2. Canary — small percentage, watch guardrails closely.
3. Gradual ramp — 5% → 25% → 50% → 100%, holding at each step.
4. Holdback — keep a few percent on the old version for a while, so you can still measure the difference after “full” rollout.

That last one is undervalued. Without a holdback you lose the counterfactual the moment you ramp to 100%, and you can’t tell whether a later metric shift came from your change or the world.

Automate rollback on guardrail breach. Manual rollback at 3am doesn’t happen fast enough.

Prompt and model versioning

Every response should be attributable to a specific prompt version, model version and retrieval index version.

log.info("generation", extra={
    "trace_id": trace_id,
    "prompt_version": PROMPT_VERSION,
    "model": MODEL_ID,           # pinned, never "latest"
    "index_version": INDEX_VERSION,
    "tokens_in": usage.input, "tokens_out": usage.output,
    "cost_usd": cost, "latency_ms": ms,
})

Never point production at an unpinned model alias. A silent provider upgrade changes behaviour with no deployment on your side, and without version stamps you cannot correlate a quality shift with it. This is the single most common preventable incident in LLM products.

Continuous evaluation

Sample production traffic and score it on a schedule — the same judge and rubric as offline, on a small daily sample.

It catches drift that a static eval set can’t: changing query distribution, degrading retrieval as the corpus grows, and provider-side model changes. Alert on the score moving, not just on errors.

Interview angle

• “Offline eval improved but the A/B test was flat. Why?” — the proxy diverged from the outcome. Common causes: eval-set distribution differs from real traffic, a latency or cost regression offset the quality gain, or the metric measured something users don’t value.
• “What signals do you use in production?” — implicit ones first: task completion, escalation, retry/rephrase, abandonment. Explicit thumbs are under 1% volume and skewed. Route the negatives into the eval set.
• “How do you validate a change before users see it?” — shadow deployment: run the candidate on live traffic, log but don’t serve. It catches real distribution, latency under load and cost, and it gives you naturally paired samples for comparison.
• “How do you A/B test an LLM feature?” — randomise per user not per request, define one primary metric and explicit guardrails on latency, cost, escalation and refusal rate, size for the higher variance of generative output, and run long enough for novelty effects to settle.
• “Why keep a holdback after rollout?” — without it you lose the counterfactual and can’t attribute later metric movements to your change versus everything else.
• “How do you avoid being surprised by a provider model upgrade?” — pin the model version, stamp every response with prompt/model/index versions, and re-run the eval suite before adopting a new version. Pointing production at an unpinned alias is how quality changes without a deploy.