Evaluation
| # | File | Covers |
|---|---|---|
| 01 | 01_why_llm_eval_is_hard.md | why accuracy doesn’t apply; the layers to measure at |
| 02 | 02_building_eval_sets.md | where cases come from, sizing, statistical honesty |
| 03 | 03_llm_as_judge.md | the biases, pairwise over scoring, validating the judge |
| 04 | 04_online_eval_and_experiments.md | shadow deployment, A/B testing, why offline and online diverge |
Retrieval-specific evaluation lives in ../09_rag_embeddings/09_agentic_rag_and_evaluation.md.
Three things to have ready
Deterministic checks before judges. Schema validity, required strings, citation presence and refusal behaviour are all free and catch most regressions. A judge is the fallback for what code cannot verify.
Pairwise beats scoring. Absolute scores drift and cluster around 7-8. Run both orderings and treat disagreement as a tie — that converts position bias into an honest tie rather than a coin flip.
85% to 88% on 100 cases is noise. The confidence interval is roughly ±7 points. Report intervals, or use paired comparison which has far more power.