LLM as judge

Using a model to score model output. Necessary, because the subjective dimensions have no other scalable measure — and unreliable in specific, known ways you’re expected to name.

When it’s the right tool

Use a judge for subjective quality: helpfulness, tone, coherence, whether an answer addresses the question. Use deterministic checks for everything else — schema, required strings, citations, refusal. See 02_building_eval_sets.md.

The rule: a judge is the fallback for what you cannot check with code. Reaching for one first is the common mistake, and it makes your eval slower, costlier and less reliable than it needed to be.

The known biases

This list is what interviewers are checking for.

Position and verbosity bias are the two to name unprompted. They’re large, well-documented, and cheaply mitigated.

Pairwise beats scoring

Absolute scores drift and cluster. “Which of these two is better” is a question models answer far more consistently.

def compare(query: str, a: str, b: str) -> str:
    """Run both orderings to cancel position bias."""
    first  = judge(query, a, b)      # returns "A" | "B" | "tie"
    second = judge(query, b, a)      # order swapped

    if first == "A" and second == "B":   # both prefer the same text
        return "a"
    if first == "B" and second == "A":
        return "b"
    return "tie"                          # disagreement = genuine tie

Treating judge disagreement across orderings as a tie is the key move: it converts position bias into an honest “too close to call” rather than a coin flip.

Pairwise is ideal for comparing two prompts or two models — exactly the decision you usually face.

Writing a rubric

Vague rubrics produce vague judgments. Be specific, give criteria, and use a small scale.

RUBRIC = """Score the ANSWER against the QUESTION and CONTEXT.

Criteria, in priority order:
1. Grounded: every factual claim appears in CONTEXT. Unsupported claims are
   disqualifying regardless of how plausible they sound.
2. Complete: addresses every part of the question.
3. Concise: no padding, no restating the question.

Score:
  2 = grounded and complete
  1 = grounded but incomplete, or complete with minor unsupported detail
  0 = contains an unsupported claim, or fails to address the question

Output JSON: {"score": 0|1|2, "reason": "<one sentence>", "unsupported": ["..."]}
"""

What makes that work:

• A 3-point scale, not 1-10. Models don’t reliably distinguish 7 from 8, and you don’t need them to.
• Priority order, so the judge knows what dominates.
• One disqualifying criterion stated plainly.
• Structured output including the reason — which lets you audit the judge, and is how you discover it’s scoring the wrong thing.

Ask for the reason before the score in the output ordering if the model generates left to right; the reasoning conditions the score rather than rationalising it.

Validate the judge

The step that gets skipped. A judge is a model with its own error rate — measure it.

# Human-label 50-100 eval outputs, then check agreement
from sklearn.metrics import cohen_kappa_score
kappa = cohen_kappa_score(human_scores, judge_scores)

Cohen’s kappa above ~0.6 is reasonable agreement; below ~0.4 means your judge is measuring something other than what you intended, and its numbers are decoration.

If agreement is poor: sharpen the rubric, switch to pairwise, use a stronger judge model, or accept that the dimension needs human review.

Re-validate when you change the rubric or the judge model. A judge is a component with a version, not a constant.

Cost and practicality

Judging is an extra model call per case, and a strong judge is expensive.

• Small scales and short rubrics keep judge output cheap.
• Batch offline — eval isn’t latency-sensitive, so use a discounted batch API.
• Cheap judge for the smoke set, strong judge for release gates.
• Cache by (rubric version, model version, output hash) so re-running a suite doesn’t re-pay for unchanged outputs.

Alternatives worth knowing

• Reference-based metrics (BLEU, ROUGE, BERTScore) — surface overlap, poorly correlated with quality for open-ended generation. Occasionally useful for translation or tightly-constrained summarisation.
• NLI / entailment models for faithfulness — smaller, cheaper and often more reliable than a general LLM judge for “is this claim supported by this passage”.
• Human review on a sample — the ground truth everything else is validated against. Expensive; sample rather than exhaust.

The entailment option is underused: for groundedness specifically, a dedicated NLI model is cheaper and better than asking a frontier model to eyeball it.

Interview angle

• “How do you evaluate subjective output quality?” — deterministic checks first for everything code can verify, then an LLM judge for what’s left, with a specific rubric and a small discrete scale. Validate the judge against human labels before trusting it.
• “What biases does an LLM judge have?” — position (prefers the first response), verbosity (prefers longer), self-preference (prefers its own family’s output), sycophancy, and formatting bias. Mitigate with order swapping, length control, a different judge model, and a rubric that names substance over style.
• “Scoring or pairwise?” — pairwise. Absolute scores drift and cluster around 7-8; relative comparison is far more consistent. Run both orderings and treat disagreement as a tie, which converts position bias into an honest tie rather than a coin flip.
• “How do you know your judge is any good?” — human-label 50-100 outputs and measure agreement, e.g. Cohen’s kappa. Below about 0.4 the judge is measuring something else and its scores are decoration.
• “Cheaper way to check groundedness?” — a dedicated NLI/entailment model on decomposed claims. Smaller, cheaper and often more reliable than a general judge for that specific question.
• “Should the judge see which answer is yours?” — no. Sycophancy and framing effects are real; keep the comparison blind and randomise ordering.