Structured Document Pipelines — Parsing, Tables, OCR, Extraction

Real RAG fails at the parsing step before it ever reaches retrieval. PDFs with tables, scanned documents, multi-column layouts — naive pdfplumber.extract_text() produces garbage. Senior interviews probe the pipeline, not the retrieval algorithm.

The problem with naive parsing

import pdfplumber
text = pdfplumber.open("invoice.pdf").pages[0].extract_text()

Often produces:

• Tables flattened into linear gibberish: “Item A 10 20 Item B 5 15 Item C…”
• Headers / footers mixed into body text.
• Multi-column layouts read left-to-right across columns: “ColA-line1 ColB-line1 ColA-line2 ColB-line2…”
• Bullet points / numbering lost.
• Mathematical formulas garbled.
• Scanned (image-only) PDFs return empty text.

Garbage in, garbage retrieval out.

The layers of a parsing pipeline

Raw input (PDF / DOCX / image / HTML)
  ↓ Format detection
  ↓ Native parsing (text + structure)
  ↓ OCR (if image / scanned)
  ↓ Layout analysis (columns, tables, figures)
  ↓ Element extraction (paragraphs, tables, captions)
  ↓ Chunking (layout-aware)
  ↓ Optional: schema-driven structured extraction (LLM)
  ↓ Embedding + storage

Each step has tool options.

Tools landscape

For most teams: unstructured.io or Docling for parsing; Textract / Document AI for OCR-heavy / forms-heavy workloads; LlamaParse when the others can’t handle complex layouts.

unstructured.io basics

from unstructured.partition.auto import partition

elements = partition(filename="report.pdf", strategy="hi_res")
for el in elements:
    print(type(el).__name__, el.text[:80])

Output:

Title:         Q4 2024 Earnings Report
NarrativeText: This quarter, we saw 15% YoY growth in...
Table:         <table HTML>
Image:         Figure 1: Revenue trend
ListItem:      • Improved customer satisfaction

Each element is typed. Chunkers can use the type to make smart decisions (don’t split tables across chunks, keep headers with their following content).

Strategies:

• fast — quick, text-only, no layout.
• hi_res — layout analysis, slower, better quality.
• ocr_only — force OCR (for image-only PDFs).
• auto — picks per page.

Docling — modern OSS alternative

IBM’s 2024 entry, Apache 2.0:

from docling.document_converter import DocumentConverter

converter = DocumentConverter()
result = converter.convert("paper.pdf")
markdown = result.document.export_to_markdown()

Strengths:

• Layout-aware (handles columns, tables, figures correctly).
• Exports to Markdown (LLM-friendly).
• Table extraction preserves structure.
• Fast (CPU-only inference for the standard model).

For text-heavy documents (papers, reports), Docling produces remarkably clean output. For forms / invoices, Textract still wins.

OCR — when native parsing fails

Scanned PDFs, images of documents:

import pytesseract
from PIL import Image

text = pytesseract.image_to_string(Image.open("scan.png"))

Or full pipeline:

import pytesseract
import pdf2image

# Convert PDF pages to images
pages = pdf2image.convert_from_path("scanned.pdf", dpi=300)
for i, page_img in enumerate(pages):
    text = pytesseract.image_to_string(page_img)
    # store / process

300 DPI is the sweet spot — higher dpi = better OCR but slower.

OCR confidence

data = pytesseract.image_to_data(img, output_type=pytesseract.Output.DICT)
for word, conf in zip(data["text"], data["conf"]):
    if int(conf) < 60:
        log.warning("Low confidence", word=word, conf=conf)

Track per-word confidence. Below 60 (out of 100) is suspicious. Critical for downstream extraction quality — surface low-confidence pages for manual review.

Cloud OCR (much better for hard cases)

AWS Textract:

import boto3
textract = boto3.client("textract")
response = textract.analyze_document(
    Document={"Bytes": pdf_bytes},
    FeatureTypes=["TABLES", "FORMS"],
)
# response includes table cells with row/col indices, form key-value pairs

Textract handles handwriting, multi-column layouts, tables. Cost: ~$15 per 1000 pages for full analysis. For forms / invoices / receipts, it’s worth it.

Table extraction

Tables are the hardest part of document parsing.

A common pattern: try fast OSS tool; on failure, fall back to LLM-vision (Claude / GPT-4V seeing the page as an image):

import base64
with open("page.png", "rb") as f:
    image_b64 = base64.b64encode(f.read()).decode()

response = anthropic.messages.create(
    model="claude-sonnet-4-5",
    max_tokens=2000,
    messages=[{
        "role": "user",
        "content": [
            {"type": "image", "source": {"type": "base64", "media_type": "image/png", "data": image_b64}},
            {"type": "text", "text": "Extract the table on this page as JSON. Each row a dict, fields named per header."},
        ],
    }],
)

Vision LLMs handle weird table layouts pdfplumber can’t. Slower and more expensive — use as fallback or for high-value tables.

Layout-aware chunking

After extraction, chunk respecting structure:

chunks = []
current = []

for element in elements:
    if element.type == "Title" and current:
        # New section — emit previous as a chunk
        chunks.append("\n".join(c.text for c in current))
        current = []
    current.append(element)

    # Don't let a chunk exceed N tokens
    if total_tokens(current) > MAX_TOKENS:
        chunks.append("\n".join(c.text for c in current))
        current = []

if current:
    chunks.append("\n".join(c.text for c in current))

vs naive char-based splitting: chunks stop at section boundaries, preserve table integrity, keep captions with figures.

Schema-driven extraction (Pydantic + LLM)

For invoices, contracts, forms — when you want structured fields, not free text:

from pydantic import BaseModel
from typing import Optional

class LineItem(BaseModel):
    description: str
    quantity: int
    unit_price: float
    total: float

class Invoice(BaseModel):
    invoice_number: str
    issue_date: str
    due_date: Optional[str]
    vendor_name: str
    customer_name: str
    line_items: list[LineItem]
    subtotal: float
    tax: float
    total: float

response = client.beta.chat.completions.parse(
    model=MODEL,   # pin the exact version in config
    messages=[
        {"role": "system", "content": "Extract invoice fields from the provided text."},
        {"role": "user", "content": document_text},
    ],
    response_format=Invoice,
)

invoice = response.choices[0].message.parsed

Structured output mode (OpenAI / Anthropic / Bedrock) enforces the schema. Output is Invoice instance directly — no parsing/retry on malformed JSON.

For Anthropic, use a tool with the schema and force tool_choice:

response = anthropic.messages.create(
    model="claude-sonnet-4-5",
    tools=[{"name": "submit_invoice", "input_schema": Invoice.model_json_schema()}],
    tool_choice={"type": "tool", "name": "submit_invoice"},
    messages=[{"role": "user", "content": document_text}],
)
invoice = Invoice.model_validate(response.content[0].input)

Multimodal: vision LLMs for full pages

For documents where layout is essential (charts, hand-drawn diagrams, complex tables), skip parsing entirely and give the LLM the page image:

response = anthropic.messages.create(
    model="claude-sonnet-4-5",
    messages=[{
        "role": "user",
        "content": [
            {"type": "image", "source": {...}},
            {"type": "text", "text": "Summarize this page, including data in the chart."},
        ],
    }],
)

Cost: more tokens (an image is ~1500-3000 tokens equivalent). Slower than text-based.

Hybrid: extract text where it works; fall back to vision for problem pages.

Evaluating extraction quality

Manually inspecting outputs at scale is impractical. Build an eval set:

1. Take 50 sample documents.
2. Manually transcribe their key fields / sections.
3. Run your pipeline; diff against the gold transcription.
4. Measure: extraction accuracy per field, missed fields, hallucinated fields.

For RAG specifically: measure end-to-end answer quality on questions whose answers live in tables / images / multi-column layouts. If accuracy drops there, the parsing is the bottleneck — not the retrieval.

Common gotchas

• PDFs with native text appear OCR’able. Always try text extraction first; only OCR on failure.
• Multi-column PDFs read wrong with naive tools. Use layout-aware (unstructured, Docling).
• Tables flatten to gibberish. Extract as structured data (rows/cells) or use vision LLM.
• Headers / footers contaminate every chunk. Detect and strip them in preprocessing.
• Page numbers, watermarks as noise. Filter.
• OCR confidence ignored. Low-confidence pages produce hallucinated retrievals downstream.
• Encoding issues (PDF / DOCX with non-Latin scripts) — test on multilingual samples.
• Large documents process slowly. Page-level parallelism via asyncio.gather + page workers.

Pipeline architecture

async def parse_pipeline(doc_url):
    # 1. Download
    raw = await download(doc_url)

    # 2. Type detection
    doc_type = detect_type(raw)

    # 3. Format-specific extraction
    if doc_type == "pdf":
        elements = await pdf_to_elements(raw, strategy="hi_res")
    elif doc_type == "docx":
        elements = await docx_to_elements(raw)
    elif doc_type == "image":
        elements = await ocr_to_elements(raw)
    else:
        raise UnsupportedType(doc_type)

    # 4. Post-processing
    elements = strip_headers_footers(elements)
    elements = resolve_tables(elements)   # structured table → dict

    # 5. Optional: schema extraction
    if doc_type == "invoice":
        invoice = await llm_extract(Invoice, elements)
        return {"raw": elements, "structured": invoice}

    # 6. Chunking
    chunks = layout_aware_chunk(elements)

    # 7. Embedding + store
    await embed_and_store(chunks)

Failures at each step should be loggable, retryable, and surface to a review queue for manual inspection.

Interview angle

• “Naive pdfplumber.extract_text() is producing garbage. Why?” — tables flatten into linear text; multi-column layouts read across columns; scanned PDFs return empty; headers/footers contaminate. Use layout-aware tools (unstructured.io, Docling) or vision LLMs.
• “Pipeline for PDFs mixing native text and scans?” — try text extraction first; if confidence low or no text, OCR via Tesseract or Textract. Combine results. Track per-page confidence; surface low-confidence pages for review.
• “How do you extract structured fields from invoices?” — schema-driven extraction with Pydantic + LLM structured-output mode. Define Invoice(BaseModel), pass to chat.completions.parse(response_format=Invoice). Output validates against the schema; no JSON-parse retries.
• “When use vision LLMs vs text-based parsing?” — when layout is essential (charts, complex tables, hand-drawn diagrams). Hybrid: text parsing for most, vision fallback for problem pages. Cost: 5-10× more tokens per page.
• “How do you handle tables in RAG?” — extract as structured (rows/cells) using camelot, table-transformer, Textract, or Docling. Pass to LLM as Markdown table or HTML; don’t let them flatten into prose. For complex tables, vision LLM as fallback.
• “How do you evaluate extraction quality?” — gold-standard hand-transcribed sample (50-100 docs); diff pipeline output; measure per-field accuracy. End-to-end: measure RAG answer quality on questions whose answers depend on parsed structures.
• “What’s unstructured.io vs Docling?” — both layout-aware parsers. unstructured.io has broader format support (DOCX, HTML, email, etc.) and a hosted service. Docling is IBM’s OSS, very good on PDFs and exports clean Markdown. Pick based on your formats + ecosystem fit.