Define the document boundary before choosing a model
Complex documents are not one data type. Contracts, purchase orders, invoices, inspection reports, and handwritten forms can contain rotated pages, stamps, tables, repeated headers, cross-page fields, and scans of uneven quality. Starting with the question of which OCR engine is most accurate often misses the larger failure modes: whether the document was classified correctly, whether its pages were assembled in the right order, and whether downstream systems know where an extracted value came from.
Begin with a processing contract for every supported document class. Define required and optional fields, repeatable sections, expected formats, cross-field relationships, and what the system must do when evidence is insufficient. Select OCR based on language coverage, handwriting needs, table density, deployment constraints, and data sensitivity. Templates and coordinate rules remain effective for stable layouts because they are predictable and inexpensive. Layout models and LLMs are more adaptable when many suppliers use changing formats, but they require stronger validation and observability.
Preserve layout instead of flattening everything into text
A plain OCR transcript discards information that later stages need. A currency label may sit beside an amount, a table header may appear only on the first page, and a note may override a value in the main section. The intermediate representation should therefore retain page numbers, text blocks, bounding boxes, reading order, table cells, merged regions, and OCR confidence. Store the original image, raw OCR output, and normalized representation separately so that corrections never erase the underlying evidence.
Image preprocessing also needs restraint. Deskewing, orientation correction, cropping, and moderate enhancement can improve scanned pages, while aggressive binarization may remove faint stamps, thin table lines, or handwriting. Multi-column pages and continued tables should not be reconstructed solely by a left-to-right text order. A safer design identifies layout regions first, then applies separate parsing strategies to headings, paragraphs, key-value areas, and tables.
- Key-value regions:Retain spatial relationships and support synonymous labels rather than relying on one exact caption.
- Tables:Preserve rows, columns, page references, and header associations instead of converting each row into unstructured text.
- Cross-page content:Mark continuation pages, repeated headers, subtotals, and footers so they are not mistaken for line items.
- Visual signals:Treat stamps, checkboxes, and signatures as separate observations rather than assuming OCR will convert them into meaningful text.
Use the LLM for semantic mapping, not unsupported decisions
LLMs are useful for interpreting label variations, resolving local context, and mapping unfamiliar layouts into a common schema. They should not receive an entire document with an open-ended instruction to produce the final answer. A more controlled pipeline uses OCR and layout analysis to create candidate regions, then asks the LLM to return a strict structure. Each field specification should state its data type, required status, allowed values, formatting rules, and evidence requirements. It should explicitly permit a missing value so the model has no reason to guess.
Attach a page number, source fragment, or bounding box to every important result. This makes extraction auditable and lets a review interface highlight the exact evidence. For tables, infer the column structure first and map rows in manageable groups rather than asking the model to reconstruct a large table in one pass. Prompts and model versions should be managed like code, with version history, representative fixtures, and regression comparisons. Sensitive documents also require a deliberate review of retention policies, processing regions, redaction, encryption, and whether private deployment is necessary.
Make validation and exceptions first-class components
An OCR confidence score is not the same as field correctness, and valid structured output does not make an LLM response trustworthy. Keep validation independent from extraction and use explainable rules for dates, amounts, currencies, tax values, identifiers, and relationships between fields. A pipeline might compare line-item totals with the stated total, verify chronological order, look up a supplier identifier in the ERP master, or flag conflicting currencies within the same document. These checks combine document evidence with the business context that a generic model does not possess.
Exceptions should have more than successful and failed states. Distinguish unreadable images, unknown document classes, missing required fields, conflicting candidates, failed business rules, and missing external reference data. Each reason needs its own retry policy, ownership, and review experience. Otherwise, technical failures and genuine business exceptions accumulate in one queue and become difficult to resolve.
- Automatic acceptance:Required fields are present, evidence is traceable, and all critical validation rules pass.
- Targeted review:Only uncertain or conflicting fields are shown, together with their locations in the source image.
- Technical reprocessing:Orientation errors, missing pages, or an unsuitable OCR path trigger a controlled retry.
- Rejection or return:The document is unsupported, unreadable, or missing information required by the business process.
Operate the pipeline as a changing production system
After launch, degradation often comes from supplier redesigns, new scanning devices, renamed fields, or changed ERP master-data rules rather than a model suddenly failing. Record the inputs and outputs of every stage, processing time, OCR and LLM versions, prompt version, validation findings, reviewer corrections, and final delivery status. Monitor results by document class, source, and field. A single aggregate success metric can conceal a serious regression affecting one layout or one critical field.
A regression set should include clean documents, poor scans, blank and duplicate pages, continued tables, similar labels, missing values, and conflicting evidence. Human corrections can guide rule improvements and future evaluation, but they should not flow into training automatically without governance. A mature document pipeline does not promise the elimination of review; it makes the automation boundary, failure reasons, and accountability explicit. When outputs also feed LINE workflows, ERP, CRM, or cloud storage, an integration team should design data contracts and exception states before treating those systems as final connectors.