ai_ml / agents orchestration / 09_llm_json_validation.md

LLM JSON Validation — Common Interview Questions and Answers

4 interview angles 7 min read source

LLM JSON Validation — Common Interview Questions and Answers

1. Why is JSON validation important when working with LLM output?

LLMs generate free-form text. Even when you ask for JSON, they can return:

  • Invalid JSON (truncation, extra commas, unescaped quotes, markdown code fences)
  • Wrong types or missing required fields
  • Hallucinated keys or values outside allowed enums

Validation ensures your application gets structured, type-safe data and fails fast with clear errors instead of crashing or behaving unpredictably downstream.

2. What are common ways to get structured JSON from an LLM?

  • Prompting: Ask explicitly for “valid JSON only” and provide a schema or example in the prompt.
  • Output format / grammar: Use constrained decoding or grammar-based decoding (e.g. JSON schema → grammar) so the model can only emit valid JSON.
  • Structured output APIs: Use provider features (e.g. OpenAI response_format: { type: "json_object" }, or structured outputs with a schema) so the model is constrained to valid JSON or to a schema.
  • Post-processing: Parse the raw string (strip markdown, fix common errors) and validate with a schema; retry or fallback on failure.

3. What is “constrained decoding” or “grammar-based decoding” for JSON?

The idea is to restrict the token set at each step so the model can only generate strings that conform to a grammar (e.g. JSON, or a specific JSON schema). The decoder uses the grammar to know which tokens are valid next (e.g. after "key": only allow value-start tokens). So the output is guaranteed to parse as valid JSON (and optionally match the schema), instead of hoping the LLM follows instructions.

Libraries/tools: Outline/outlines, llama.cpp grammar, Guidance, etc.

4. How do you handle LLM output that is wrapped in markdown code blocks?

Strip the wrapper before parsing:

import re
import json

def extract_json(text: str) -> str:
    # Remove ```json ... ``` or ``` ... ```
    match = re.search(r"```(?:json)?\s*([\s\S]*?)\s*```", text)
    if match:
        return match.group(1).strip()
    return text.strip()

raw = model_output  # e.g. "```json\n{\"a\": 1}\n```"
json_str = extract_json(raw)
data = json.loads(json_str)

Then validate the parsed data against your schema (e.g. Pydantic, jsonschema).

5. What are common JSON errors in LLM output and how do you fix them?

  • Trailing commas: {"a": 1,} — remove or replace ,} / ,] before parsing, or use a lenient parser.
  • Unescaped quotes in strings: {"say": "He said "hi""} — hard to fix reliably; ask for escaped output or constrain decoding.
  • Truncation: Incomplete JSON — use max_tokens and stop sequences wisely; detect incomplete parse and retry or fallback.
  • Wrong types: Number instead of string, etc. — validate with a schema (Pydantic, jsonschema) and coerce or reject.
  • Markdown/explanation around JSON: Use regex or heuristics to extract the JSON block, then parse and validate.

6. How do you use Pydantic to validate LLM JSON output?

Define a Pydantic model for the expected shape, parse the string to a dict, then validate:

import json
from pydantic import BaseModel, ValidationError

class Person(BaseModel):
    name: str
    age: int
    tags: list[str] = []

def parse_llm_json(raw: str) -> Person:
    data = json.loads(raw)  # or extract_json(raw) first
    return Person.model_validate(data)

Catch json.JSONDecodeError for parse errors and ValidationError for schema violations. Use model_validate_json(raw) to parse and validate in one step if the string is pure JSON.

7. What is the difference between validating “any JSON” and “JSON that matches a schema”?

  • Any valid JSON: Only check that the string parses (e.g. json.loads). You get a dict/list but no guarantees on keys, types, or values.
  • Schema match: Check that the parsed object conforms to a schema (Pydantic model, JSON Schema, etc.): required fields, types, enums, ranges. Your app can rely on structure and types; invalid LLM output is rejected or corrected.

For LLM integration, schema validation is usually required so the rest of your pipeline is type-safe.

8. How do you use OpenAI’s structured output (e.g. JSON schema) for LLMs?

OpenAI supports structured outputs: you pass a JSON schema (or use a Pydantic model that is converted to one), and the API constrains the response to that schema. You get valid JSON that matches the schema without writing parsing/validation yourself. Similar ideas exist for other providers (e.g. tool/function calling with a defined response shape). Check the provider docs for the exact parameter (e.g. response_format, structured output mode).

9. What is a good retry strategy when LLM JSON validation fails?

  • Retry with same prompt: Sometimes the model succeeds on retry (non-deterministic).
  • Retry with error feedback: Include the validation error in a follow-up message (e.g. “Your previous response had this error: … Please return valid JSON.”) so the model can self-correct.
  • Limit retries: e.g. 1–3 attempts to avoid cost and latency.
  • Fallback: Default values, optional fields, or a safe “invalid” result so the app doesn’t crash.
  • Log failures for monitoring and prompt/schema improvements.

10. How do you validate that required fields are present and correctly typed?

Use a schema validator:

  • Pydantic: Define a BaseModel with required fields (no default) and the right types; call Model.model_validate(data). Pydantic checks presence and types and can coerce (e.g. string to int) if configured.
  • JSON Schema: Use a schema with required and type (and properties) and validate with a library like jsonschema.validate().

Missing or wrong-type fields raise a clear error; you can catch it and retry or return a default.

11. How do you handle optional fields or multiple possible structures from an LLM?

  • Optional fields: In Pydantic use Optional[T] = None or T | None = None; in JSON Schema use optional properties (not in required).
  • Multiple structures: Use a discriminated union (e.g. Pydantic Field(discriminator='type') with different models) or a union of models and validate; only one variant should match. Alternatively ask the LLM for a single structure and validate that.

12. What is “output parsing” in the context of LLM libraries (e.g. LangChain)?

Output parsing means taking the raw LLM string and turning it into a structured object. Parsers often:

  • Strip markdown or extract the JSON part.
  • Parse JSON (or XML, etc.).
  • Validate against a schema (e.g. Pydantic) and raise or return a typed object.

So “output parser” = parse + validate + typed result. Some libraries have built-in parsers (e.g. PydanticOutputParser) that also format the schema into the prompt so the model is more likely to comply.

13. Why might you use both a prompt and a schema for LLM JSON?

  • Prompt: Tells the model what to return (structure, meaning, examples), improving correctness and reducing irrelevant text.
  • Schema: Defines the exact contract (types, required fields, enums). Validation catches when the model drifts.

Together: the prompt improves behavior; the schema guarantees valid, typed data. With structured output APIs, the schema can also constrain decoding so invalid JSON is not generated.

14. How do you secure or sanitize JSON from an LLM before using it in your app?

  • Validate schema: Reject unknown keys or wrong types if your schema is strict (extra='forbid' in Pydantic, etc.).
  • Allowlist keys: Only use keys you expect; ignore or drop the rest.
  • Sanitize strings: If you render LLM output (e.g. in HTML), escape it to prevent XSS; limit length to prevent DoS.
  • Don’t eval: Parse with json.loads (or equivalent); never eval() LLM output.
  • Rate limits and quotas: Limit how often you call the LLM and how much output you accept to reduce abuse and cost.

15. What is the trade-off between strict JSON schema and flexibility with LLM output?

  • Strict schema: Guarantees shape and types; easier to use in code; fewer bugs. Risk: valid answers may be rejected if the model uses a different but acceptable format (e.g. extra field, different enum string), and you may need more retries or prompt tuning.
  • Flexible / loose schema: Accepts more variation (optional fields, extra keys). Less rejection, but your code must handle many cases and you lose type safety.

Practical approach: start with a strict schema for the minimal structure you need; relax only specific fields (e.g. optional, allow extra) where the LLM consistently varies in a harmless way.

Interview angle

  • “How do you get reliable structured output?” - constrained decoding against a schema where the provider supports it, so invalid output is impossible rather than detected. Otherwise validate with Pydantic and retry feeding the validation error back, which usually succeeds on the second attempt.
  • “Why prefer Literal types over free-text fields?” - the model cannot invent a category that isn’t in the enum. It converts a whole class of downstream errors into a schema constraint.
  • “What do you do when validation fails repeatedly?” - cap the retries and fail explicitly rather than looping. Persistent failure usually means the schema is ambiguous or too complex, not that the model is uncooperative - simplify and split it.
  • “Temperature for structured output?” - zero. Non-determinism in a path whose output you parse is a bug, not creativity.
Previous
Function Calling, Tool Use, and Structured Output