Fluent Is Not the Same as Faithful
Large language models are good at compressing meaning, but they are not naturally designed to preserve every exact detail. A model may turn “NT$500,000, excluding tax” into “about NT$500,000,” shorten “payment is due within ten business days after acceptance” to “payment follows acceptance,” or omit the person who must approve an action. The result sounds reasonable, yet the tax basis, deadline, or accountability chain has changed. That is unacceptable when a summary informs procurement, legal review, finance, incident response, or project execution.
The first engineering task is therefore not prompt tuning. It is defining a set of protected facts that must survive summarization. These commonly include amounts, currencies, tax treatment, quantities, ratios, dates, time zones, versions, thresholds, exceptions, obligations, prohibitions, deliverables, owners, approvers, and escalation contacts. A number must also retain its role. “Thirty days” is not actionable unless the system preserves whether it is a payment period, warranty notification window, or termination notice, together with its unit and triggering event.
Protection levels should reflect business risk. An informal project update can tolerate more paraphrasing than a contract, quotation, SLA, audit record, or incident report. When high-risk sources conflict, the system should surface the discrepancy for review instead of selecting the statement that merely appears more plausible.
Model Claims, Evidence, and Accountability
A dependable architecture does not move directly from source documents to polished prose. It first creates an intermediate record of important claims, then asks the model to write from that record. Every claim should retain a source locator such as a document name, page, paragraph, table row, or message timestamp. This allows validation to compare structured fields rather than asking another model whether an entire paragraph “looks correct.”
- Numbers: Store the original text, normalized value, unit, currency, tax status, applicable period, and comparison operator. “No more than” cannot be reduced to a bare value.
- Terms: Capture the obligated party, required action, precondition, deadline, exception, consequence, and governing document version. Preserve distinctions such as must, may, and must not.
- Ownership: Record who performs the work, who approves it, who is consulted, who must be informed, and whether the named parties have confirmed the assignment.
- Evidence: Attach a precise locator and a short source excerpt to each claim. If several passages support or contradict it, keep all of them and their precedence.
- Status: Distinguish proposed, agreed, pending confirmation, completed, and withdrawn items so that a discussion option does not become a recorded decision.
This structure exposes a practical trade-off. Extractive summaries stay close to the source but can be tiring to read; free paraphrasing improves readability while increasing the chance of losing qualifications. A useful compromise is concise narrative prose accompanied by fixed-format lists of figures, terms, and owners, each linked to its evidence.
Separate Extraction, Generation, and Validation
A robust pipeline has three stages. First, extract candidate facts with both deterministic tools and a language model. Dates, monetary values, percentages, version strings, and identifiers are good candidates for parsers and pattern-based checks. Conditions, negations, exceptions, and cross-sentence ownership relationships benefit from model interpretation. Rules reduce the chance of missing formatted values; models provide context. Neither should replace the other.
Second, generate the summary from both the source material and the protected-fact record. The generation instructions should prohibit merging unlike units, silently converting values, removing exceptions, or assigning an owner when the source is ambiguous. If two sources provide different deadlines, the output should present both values with their evidence and mark the conflict. It should not guess which one is current.
Third, validate the generated result. Re-extract numbers, conditions, and ownership statements from the summary and compare them with the intermediate record. Check currency, units, dates, comparison operators, modality, subject, and status. A model can help assess semantic consistency, but exact fields such as amounts, dates, and IDs should use deterministic comparisons whenever possible. Failed validation should stop automatic publication, record the reason, and show the reviewer the relevant source passages.
Evaluate Actionability, Not Just Similarity
Test sets should represent the document problems teams actually encounter: a table disagreeing with the narrative, an attachment superseding the main document, people with identical names in different departments, relative dates without a clear reference point, clauses with nested exceptions, and action items that were proposed and later withdrawn. Evaluation questions should be operational: Does every amount retain its currency and tax basis? Does every deadline include a trigger? Does every action have an owner and status? A generic similarity score will not answer those questions.
Production systems also need versioning and auditability. Retain a hash of the source, extracted facts, model and prompt versions, validation results, and human corrections so the summary can be reconstructed when challenged. Access controls must follow the source systems as well. A user should not gain access to restricted contract or personnel information merely because a summary combines multiple repositories.
The most important policy is simple: do not complete what the evidence cannot confirm. A summary that clearly marks missing values, conflicting terms, and unconfirmed owners is more trustworthy than one that sounds complete while changing the commercial meaning. When this capability is integrated with enterprise RAG, LINE, ERP, or CRM workflows, the data model, validation gates, and human review experience deserve as much attention as the generation model itself.