Domain 4 — Prompt Engineering & Structured Output (20%)

System prompt design, the Claude API’s output-control mechanisms, and reliable structured data extraction with validation.

Subtopics to cover

System prompt engineering — principles vs. conditionals, few-shot, prompt dilution, clarifying questions.
API fundamentals & output control — stateless Messages API, structured outputs, tool use, prefill, tool_choice.
Structured data extraction — schema design, reducing fabrication, provenance.
Validation and confidence — semantic validation, confidence thresholds, human review.
Batch processing.

The system prompt is sent on every request (no persistent memory), and attention to it weakens as the conversation grows.
Use XML-style sections for organization and salience.
General principles for judgment-heavy behavior; explicit conditionals for safety-critical triggers.
Few-shot examples outperform prose instructions — especially for edge cases and format.
Prompt dilution — endless conditionals and long prose bury the rules that matter. Keep prompts concise with salient sections; reinforce at natural breakpoints.
Clarifying questions for ambiguous cases; proceed with stated assumptions for low-risk work. Prefer one focused question over a list. Surface conflicting preferences rather than averaging them.

The Messages API is stateless — every request carries the full context.
System prompts go in the system parameter, not a message role.
Structured outputs via output_config.format; tool use with strict parameter enforcement.
tool_choice: auto (model decides), any (must use some tool), a specific tool name, or none.
Tool definitions cost tokens and consume context budget.
Partial assistant prefill controls output format.
Schema validation is an output-control mechanism — prefer tool use over “please return JSON” in text.

Schema-backed output is the foundation of reliable extraction.
Optional/nullable fields reduce fabrication pressure — a required field the model can’t fill invites a hallucination.
Absence semantics: optional fields, empty arrays, null, or an unclear enum value — give the model a way to say “not present.”
Few-shot examples for edge cases and format consistency; escape hatches for long-tail categories.
Provenance fields — source location, exact quotes; handle amendments with effective dates.

Semantic / domain validation goes beyond JSON Schema (a date can be schema-valid but logically wrong).
On failure, send correction requests with the exact validation errors.
Reconciliation fields for internally inconsistent source data.
Staged extraction for long, scattered documents — don’t push a 200-page doc through one call.
Calibrated confidence thresholds per document type and field; validate the automation threshold with stratified sampling, not aggregate accuracy alone.
Feedback loops for recurring error patterns.

Use batch strategies for high-volume extraction; don’t route every long document through a single synchronous call.

Using IMPORTANT / NEVER in caps as a reliability mechanism.
Adding endless conditionals; hiding key rules in long prose.
Forcing JSON via prompt text instead of tool use / structured outputs.
Assuming Claude has persistent memory; treating session_id as model memory.
Confusing tool_choice: "auto" with required tool use.
Ignoring tool-definition token cost.
Treating valid JSON as correct data; confusing schema compliance with source truth.
Making absent source fields required; using strict enums with no escape hatch.
Relying only on aggregate accuracy; sending all long documents through one extraction call.

“Extraction keeps hallucinating a phone value that isn’t in the document — what’s the schema fix?” — Make phone optional/nullable so the model can legitimately say it’s absent instead of inventing one.
“You need guaranteed JSON output — prompt instruction or tool use?” — Tool use / structured outputs with strict parameter enforcement. Prompt text doesn’t guarantee format.
“The model ignores a rule late in a long conversation — why, and what helps?” — System-prompt attention weakens as context grows; reinforce the rule at natural breakpoints and keep the prompt concise and salient.
“Extraction returns valid JSON — are you done?” — No. Schema-valid ≠ correct. Add semantic validation and provenance checks against the source.

“How do you get reliably structured output?” - tool use or a schema-constrained response, not a request to reply in JSON. Parsing free text and hoping is the pattern that fails at scale, and retry-on-parse-error is a symptom of not using the constrained path.
“What actually improves a prompt?” - being specific about the task, giving examples that cover the edge cases rather than the easy ones, and stating what to do when the input is ambiguous. Politeness formulas and role-play preambles are mostly noise.
“Where does system prompt content belong versus per-request content?” - stable instructions in the system prompt so they cache; variable data per request. Getting this backwards defeats prompt caching and multiplies cost.
“How do you evaluate a prompt change?” - against a fixed evaluation set with a metric, not by trying three examples. Prompt changes regress silently, which is why the eval suite is part of the deliverable. See ../../ai_ml/13_evaluation/.