Guardrails and output validation

The layer between the model and everything downstream. The design principle: validate deterministically wherever you can, and use a model to check only what code cannot.

Input and output guardrails

Input guardrails are cheap and prevent waste. Output guardrails prevent harm. You need both, and output is where the real risk sits.

Structured output first

The single most effective guardrail is not accepting free text at all.

class TicketTriage(BaseModel):
    category: Literal["billing", "technical", "account", "other"]
    urgency: Literal["low", "medium", "high"]
    summary: str = Field(max_length=200)
    needs_human: bool
    confidence: float = Field(ge=0, le=1)

result = client.chat.completions.parse(
    model=MODEL, messages=msgs, response_format=TicketTriage,
)

Literal types mean the model cannot return an invented category. Constrained decoding enforces the grammar during generation rather than validating afterwards, so the failure mode disappears instead of being caught.

Where the provider supports it, use native structured output. Where it doesn’t, validate and retry:

for attempt in range(3):
    raw = llm(messages)
    try:
        return TicketTriage.model_validate_json(raw)
    except ValidationError as e:
        messages.append({"role": "user",
                         "content": f"Invalid output: {e}. Return valid JSON only."})
raise OutputValidationError

Feeding the validation error back usually works on the second attempt. See ../10_agents_orchestration/09_llm_json_validation.md.

Groundedness checking

For anything RAG-based, the failure that matters is a fluent claim with no support.

def check_grounded(answer: str, context: list[str]) -> list[str]:
    """Return claims not entailed by the context."""
    claims = decompose_into_claims(answer)
    return [c for c in claims if not entails(context, c)]

Decompose the answer into atomic claims and check each against the retrieved passages. A dedicated NLI/entailment model is cheaper and often more reliable here than asking a frontier model to eyeball it.

What to do on failure: regenerate with stricter instructions, strip the unsupported sentence, or return “I don’t have information about X” — which is nearly always better than shipping a plausible fabrication.

Require citations and verify them mechanically: every citation must reference a chunk that was actually retrieved. Models invent citations, and a citation pointing at a document that wasn’t in context is trivially detectable.

Layering, cheapest first

1. Deterministic  - schema, regex, length, allowlists         ~free
2. Classifier     - small model for PII, toxicity, topic      ~ms
3. LLM judge      - groundedness, tone, nuance                ~100s of ms
4. Human          - high-stakes, low-confidence               minutes

Run cheap checks first and short-circuit. Never spend a judge call on output that failed schema validation.

The latency problem

Output guardrails conflict with streaming: you cannot validate text you haven’t finished generating, but users want tokens immediately.

For anything with side effects, the last row is right: let the text stream to the user, but gate the downstream action on validation completing.

Frameworks

NeMo Guardrails, Guardrails AI, Llama Guard and provider-native moderation endpoints all exist. They’re useful, and none removes the need to know what you’re checking for.

The pragmatic position: use Pydantic for structure (which you already have), a small classifier for PII and toxicity, and a judge for groundedness. Reach for a framework when you need policy managed separately from code, or an auditable rule set for compliance.

Fail open or closed

An explicit decision per check, not a default.

And decide what a blocked response looks like. “I can’t help with that” with no explanation is a bad experience; explaining precisely why can leak the policy. Usually: a generic message to the user, a detailed reason in the logs.

Interview angle

• “How do you stop an LLM returning garbage to a downstream system?” — don’t accept free text. Constrained structured output with Literal types makes invalid categories impossible rather than detectable, and validation with error feedback handles the rest.
• “How do you check an answer is grounded?” — decompose into atomic claims and check entailment against the retrieved context, ideally with a dedicated NLI model rather than a general judge. Require citations and verify mechanically that each references a chunk actually retrieved.
• “How do you layer guardrails?” — cheapest first: deterministic checks, then a small classifier, then an LLM judge, then a human for high-stakes low-confidence cases. Short-circuit early so you never pay for a judge on output that failed schema validation.
• “How do output guardrails work with streaming?” — they conflict. For human-facing text, stream and validate in parallel; for anything with side effects, let text stream but gate the downstream action on validation completing.
• “Fail open or closed?” — per check. Closed for PII, unsafe content, schema violations feeding a system, and unauthorized actions. Open with logging for stylistic checks, where blocking costs more than it saves.
• “Do you need a guardrails framework?” — not initially. Pydantic for structure, a small classifier for PII and toxicity, and a judge for groundedness cover most needs. Frameworks earn their place when policy must live outside code or be auditable for compliance.