Why evaluating LLM systems is hard
Classical ML has a test set and a metric. LLM applications have neither, by default. Understanding why is the setup for every other answer in this folder.
What breaks
No single correct answer. For “summarise this ticket” there are many good outputs and no string to compare against. Exact match is useless; BLEU and ROUGE measure surface overlap, which correlates poorly with whether the summary is actually good.
Non-determinism. The same input can produce different output. Even at temperature 0, GPU batching changes floating-point reduction order. Any test asserting an exact string is flaky by construction.
The failure modes are semantic. A response can be fluent, well-formatted, on-topic and wrong. Nothing in the output signals it. Compare with a classifier, where a wrong label is unambiguous.
Quality is multi-dimensional. Correct, grounded, appropriately formatted, appropriately refusing, appropriately concise, cheap, fast. A single number can’t express that, and optimising one dimension often degrades another.
The model changes under you. A provider upgrade shifts behaviour without any change on your side. This is the one people miss: pin model versions and re-run evals on upgrade, or you have no idea when your quality moved.
The layers
Evaluate at the level you can act on. A single end-to-end number tells you something broke, not what.
| Layer | Measures | Method |
|---|---|---|
| Unit — component | does retrieval find the right chunk? does the parser handle this shape? | deterministic assertions |
| Task | does the system produce a good answer for this input? | eval set + judge or human |
| Trajectory | did the agent take a sensible path? | assertions on tool-call sequences |
| System | latency, cost, error rate, refusal rate | ordinary telemetry |
| Business | resolution rate, escalations, satisfaction | A/B test |
The most useful diagnostic split, for RAG: measure recall@k separately from answer quality. If the right chunk isn’t retrieved, no amount of prompt work fixes the answer, and you’ve been tuning the wrong stage. See ../09_rag_embeddings/09_agentic_rag_and_evaluation.md.
What to measure
| Dimension | Question | How |
|---|---|---|
| Correctness | is the answer right? | eval set with expected outputs, or a judge |
| Faithfulness | is every claim supported by the provided context? | claim decomposition + entailment check |
| Relevance | does it answer this question? | judge |
| Format compliance | does it parse / match the schema? | deterministic — just validate |
| Safety | does it refuse what it should, and not what it shouldn’t? | adversarial set |
| Cost | tokens per request, cost per resolved task | telemetry |
| Latency | p50/p95, time to first token | telemetry |
Format compliance and cost are deterministic — measure them properly and stop guessing. A schema validation pass rate is a real metric with no judge required, and it catches a large share of production failures.
Cost per resolved task is often the metric that actually decides things, and it’s the one most eval frameworks omit.
Refusals cut both ways
Two failure directions, and most teams measure only one:
- Over-refusal — declining legitimate requests. Invisible unless you test for it, and it silently destroys usefulness.
- Under-refusal — complying with things it shouldn’t.
Track both rates. A model tuned to be safe can become useless in exactly the ways nobody notices until users leave.
The honest position on benchmarks
Public benchmarks tell you little about your application. They’re contaminated (test sets leak into pretraining data), they measure generic capability rather than your task, and they’re optimised against by model providers.
A hundred examples from your own domain beats any public leaderboard for deciding whether a model works for you. Saying that plainly is a good signal.
What good looks like
A production LLM system has:
- A curated eval set from real usage, versioned in the repo.
- Deterministic checks where possible — schema, format, required citations, forbidden content.
- A judge for the subjective dimensions, validated against human labels.
- Regression gates in CI on prompt, model, or retrieval changes.
- Online metrics — the business outcome, not the proxy.
- A feedback loop turning production failures into new eval cases.
Point 6 is what separates a system that improves from one that plateaus. Every user complaint should become a test case.
Interview angle
- “Why can’t you just use accuracy for an LLM feature?” — there’s usually no single correct output, generation is non-deterministic, and failures are semantic rather than structural. Quality is also multi-dimensional, so one number hides the trade-offs.
- “How would you evaluate a RAG chatbot?” — split the layers. Retrieval recall@k against a golden set, faithfulness of the answer against retrieved context, format compliance deterministically, then online resolution rate. Measuring end to end only tells you something broke.
- “What do you measure that most people don’t?” — format compliance and cost per resolved task, both deterministic and both decision-relevant. Plus over-refusal, which is invisible unless you deliberately test for it.
- “Your provider upgraded the model. What do you do?” — you should have pinned the version. Re-run the eval suite against the new version before switching, and compare on your own set rather than trusting the vendor’s benchmark deltas.
- “Are public benchmarks useful?” — for coarse model selection, marginally. They’re contaminated and optimised against, and they don’t measure your task. A hundred domain examples of your own is worth more.
- “How do you keep an eval set from going stale?” — feed production failures back into it. Every complaint or escalation becomes a case, so the suite tracks how the system actually fails rather than how you imagined it would.