Clinical document intelligence: protocol to structured workflow

The system design behind the clinical-trials vacancy: an AI protocol engine converting 100-400+ page trial documents into structured workflows. This is a RAG and extraction problem with regulatory constraints, and it is the highest-value thing to be able to design out loud for that role.

The problem shape

Input: a long PDF written for humans — a trial protocol with inclusion and exclusion criteria, a schedule of assessments, dosing rules, endpoints and safety reporting requirements.

Output: structured, verifiable objects — visits, procedures per visit, eligibility rules, timing windows — that a site can execute and a system can enforce.

The hard part is not summarisation. It is extraction that is complete, structured, traceable and reviewable, because a missed criterion or a wrong visit window is a protocol deviation with regulatory consequences.

Pipeline

PDF -> layout-aware parse -> section segmentation -> structured extraction -> validation -> human review -> workflow
                                     |                        |
                              retrieval index          citations back to page/section

Parsing is where quality is won or lost. Protocols are full of multi-page tables (the schedule of assessments is often the single most important table in the document), nested numbering, headers and footers, and cross-references. Naive pdfminer text extraction destroys table structure and silently interleaves columns. Use a layout-aware parser that preserves tables, and evaluate the parser separately from the model — a bad parse cannot be fixed downstream. See ../../ai_ml/09_rag_embeddings/07_document_parsing_pipelines.md.

Chunk on document structure, not character count. Protocols have a section hierarchy; splitting on it and keeping the heading path in the chunk metadata is what lets a retrieved chunk be cited as “Section 6.2.1, page 84”. Fixed-size chunking across a criteria list splits it mid-list and produces incomplete extraction.

Extract with a schema, not free text. Define Pydantic models for each target object and use constrained/tool-based generation so the model must return that shape. Parsing prose replies is the pattern that fails at scale.

class EligibilityCriterion(BaseModel):
    kind: Literal["inclusion", "exclusion"]
    text: str
    structured: CriterionExpr | None       # parsed form, when confidently derivable
    source_section: str
    source_page: int
    confidence: float

Note what the model carries: the source section and page. Every extracted field must point back to where it came from. In this domain an answer without a citation is unusable, because a human has to verify it.

Validate before it reaches a human. Schema validation is the floor; the useful layer is cross-checks — does every visit in the schedule table appear in the narrative, do the timing windows overlap, does every referenced assessment exist. These catch extraction errors that look plausible individually.

Design for review, not autonomy. The realistic product is a first pass a clinical reviewer confirms or corrects, with each item showing its source. Corrections are the evaluation data for the next iteration. Treating this as a fully automated pipeline is both a product mistake and a regulatory one — see 03_gxp_and_validation.md.

Retrieval for long regulated documents

• Hybrid search. Protocols are dense with exact terms — drug names, criteria codes, visit labels — where BM25 beats embeddings. Combine with dense retrieval and fuse with RRF. See ../../ai_ml/09_rag_embeddings/08_hybrid_search_and_reranking.md.
• Metadata filtering first. Scope to the document, version and section before ranking. A query about dosing should never retrieve from the safety-reporting appendix.
• Rerank the top candidates with a cross-encoder; the precision gain matters more than usual when a human will read every result.
• Version pinning. Protocols are amended. Every extraction records the protocol version it came from, and an amendment triggers re-extraction with a diff against the previous version — “what changed and what does that mean for sites” is a genuine product feature.

Storage

Postgres with pgvector handles this comfortably at typical trial scale — thousands of documents, not billions of vectors — and keeping the vectors next to the relational data removes an entire class of consistency problem. Use HNSW indexing, and filter on metadata columns in the same query. Reach for a dedicated vector database when scale or multi-tenancy genuinely demands it, not by default. See ../../ai_ml/09_rag_embeddings/04_vector_databases.md.

Store the document, the parse, the chunks, the extractions and the human corrections as separate durable artefacts. Reprocessing a protocol with an improved parser should not lose the reviewer’s decisions.

Cost and performance

A 400-page protocol is roughly 200k-400k tokens. Sending it whole to a frontier model per query is not viable.

• Extract once, query many. Do the expensive structured extraction as a batch job at ingest; interactive queries hit the extracted objects and the retrieval index.
• Route by difficulty. A cheap model for classification and section routing, a frontier model for the extraction that needs reasoning.
• Cache the stable prefix. Instructions and schema in the cached portion of the prompt, variable content after it.
• Batch overnight. Ingest is not latency-sensitive; use batch APIs where the provider offers a discount.

Report cost per document, not per token. It is the number the business cares about and the one that makes the routing argument concrete. See ../../ai_ml/08_inference_serving/.

Evaluation

Without this, you cannot tell whether a prompt change helped.

• A golden set of protocols with expert-verified extractions, including deliberately awkward ones.
• Per-field metrics: recall on criteria (missing one is the expensive failure), precision on visits, exact-match on timing windows.
• Citation validity: does the cited section actually contain the claim. This catches confident fabrication better than answer-quality scoring.
• A CI gate so a prompt, model or chunking change cannot ship without running the suite.

See ../../ai_ml/13_evaluation/.

Interview angle

• “How would you turn a 300-page protocol into structured workflows?” - layout-aware parse preserving tables, structure-based chunking with the heading path retained, schema-constrained extraction into Pydantic models with page and section provenance, automated cross-validation, then human review. Extraction happens once at ingest; queries hit the extracted objects.
• “Why not just put the document in the context window?” - cost and reliability. 400 pages is several hundred thousand tokens per query, and recall over a very long context is not uniform. Extract once, index, and retrieve.
• “What is the failure mode you worry about most?” - a missed inclusion or exclusion criterion. It is a silent failure - the output looks complete - so you measure recall per field against a golden set rather than judging overall answer quality.
• “Why does every extraction need a citation?” - because a human must verify it, and in a regulated setting you must be able to show where a value came from. An uncited extraction cannot be reviewed efficiently or defended in an audit.
• “Postgres with pgvector, or a dedicated vector database?” - pgvector at this scale. Thousands of documents is well within its range, and keeping vectors alongside the relational data avoids syncing two stores. Move only when scale or isolation requirements force it.
• “How do you handle a protocol amendment?” - version every document and every extraction, re-extract on amendment, and diff against the previous version. Surfacing what changed is a feature, not just bookkeeping.