Building eval sets

The unglamorous work that determines whether everything else is meaningful. A hundred good examples beats any framework.

Where cases come from

Start with production logs. Sample real queries, stratified across intents and difficulty, not just the head. Synthetic-only eval sets test the system against questions you imagined rather than questions people ask, and they systematically miss the messy ones.

What a case looks like

class EvalCase(BaseModel):
    id: str
    query: str
    context: dict = {}                 # user state, conversation history
    expected: str | None = None        # for deterministic checks
    expected_chunks: list[str] = []    # for retrieval recall
    must_contain: list[str] = []       # required facts
    must_not_contain: list[str] = []   # forbidden content
    should_refuse: bool = False
    rubric: str | None = None          # for a judge
    tags: list[str] = []               # intent, difficulty, source

Tags matter more than they look: they let you report per-segment rather than one aggregate, which is how you notice the system got worse for one intent while the average improved.

Coverage that catches real failures

Deliberately include:

• The head — the common cases, weighted as they occur.
• The tail — rare but important intents.
• Ambiguous queries where the right behaviour is to ask a clarifying question.
• Unanswerable queries — the corpus genuinely doesn’t contain the answer. Tests whether the system says so or confabulates. This category is the one most often missing, and it’s where trust is lost.
• Adversarial — injection attempts, jailbreaks, off-topic requests.
• Multi-turn — follow-ups that only make sense in context (“what about the second one?”).
• Long inputs at realistic lengths, not toy ones.

Sizing

100-200 is the sweet spot. Enough that a few-point change isn’t noise, small enough to run in minutes for a few dollars.

Run the smoke set on every change and the full set before merging or upgrading a model.

Deterministic checks first

Before reaching for a judge, extract everything checkable:

def deterministic_checks(case: EvalCase, output: str) -> dict[str, bool]:
    return {
        "parses": try_parse_json(output) is not None,
        "schema_valid": validate_schema(output),
        "contains_required": all(s in output for s in case.must_contain),
        "avoids_forbidden": not any(s in output for s in case.must_not_contain),
        "has_citation": bool(CITATION_RE.search(output)),
        "refused": is_refusal(output) == case.should_refuse,
        "within_length": len(output) < MAX_LEN,
    }

These are free, fast, deterministic and catch a large share of real regressions. A judge is for what’s left.

Statistical honesty

from scipy.stats import beta

def score_ci(passes: int, n: int, conf: float = 0.95) -> tuple[float, float]:
    """Wilson-style interval via the beta distribution. Report this, not a bare rate."""
    lo = beta.ppf((1 - conf) / 2, passes, n - passes + 1) if passes else 0.0
    hi = beta.ppf(1 - (1 - conf) / 2, passes + 1, n - passes) if passes < n else 1.0
    return lo, hi

On 100 cases, an 85% pass rate has a confidence interval of roughly ±7 points. An 85% → 88% “improvement” is noise. Reporting a bare percentage without an interval is how teams ship changes that did nothing.

If you need to detect small differences, either grow the set or use paired comparison — run both variants on the same cases and count wins, losses and ties, which has far more statistical power than comparing two independent rates.

Versioning

The eval set lives in the repo, in version control, alongside the code.

• Cases are append-mostly. Deleting a case because it started failing is how a suite stops catching regressions.
• Record the model version with every result. “Score dropped” is meaningless without knowing what changed.
• Store outputs, not just scores. When something regresses you want to read what it actually produced, and re-scoring old outputs under a new rubric is often useful.

The feedback loop

production failure -> triage -> add as eval case -> fix -> case passes
                                       |
                                 stays in the suite forever

This is the mechanism that makes a system improve. A complaint that doesn’t become a test case will recur.

Weekly triage of production failures into the eval set costs an hour and compounds. It’s a process answer rather than a technical one, and it lands well in interviews for exactly that reason.

Interview angle

• “How do you build an eval set for an LLM feature?” — sample real production queries stratified by intent and difficulty, add known failures from escalations, and deliberately include unanswerable and adversarial cases. 100-200 cases is the practical working size.
• “How many examples do you need?” — enough that your effect size exceeds the confidence interval. At 100 cases the interval is around ±7 points, so small differences need paired comparison or a bigger set.
• “What do you check without a judge?” — parseability, schema validity, required and forbidden strings, citation presence, refusal behaviour, length. Deterministic, free, and they catch most real regressions.
• “Your eval score went from 85% to 88%. Ship it?” — not on that alone. On 100 cases that’s within noise. Run a paired comparison on the same cases, or expand the set.
• “What’s the most commonly missing category in eval sets?” — unanswerable queries. Without them you never measure whether the system admits ignorance or confabulates, which is where user trust actually breaks.
• “How does an eval set stay relevant?” — triage production failures into it regularly. Cases are append-mostly; removing a failing case to make the suite green defeats the purpose.