Treat the context window as a budget, not a container
A common mistake in long-document Q and A is to treat the model's maximum token limit as space that should be filled. In production, the system prompt, conversation history, user question, retrieved passages, citation metadata, and generated answer all compete for that space. Adding more document text increases latency and cost, but it can also reduce answer quality. Irrelevant passages distract the model, while repeated language from different sections or document versions makes it easier to cite the wrong source.
Start by defining a request-level budget. Reserve enough tokens for a complete answer and its citations, set a limit for conversational context, and allocate the remainder to evidence. The allocation should vary by task. A question about one policy clause usually benefits from a few precise passages. A comparison across chapters, a timeline, or a synthesis across several documents needs broader coverage and may require multiple retrieval and generation stages instead of one oversized prompt.
Chunk around answerable units of meaning
Fixed-length chunks are easy to implement, but they often separate a heading from its explanation, a table from its notes, or a rule from its exceptions. A stronger pipeline first follows the document's structure and then applies size limits within that structure. Preserve section paths, paragraph boundaries, lists, table rows, and page references where possible. Attach metadata such as document title, version, effective date, access scope, and source location. Metadata improves filtering and gives the answer a traceable basis.
Overlap can protect information near chunk boundaries, but excessive overlap enlarges the index and sends duplicate evidence into the prompt. For hierarchical documents, consider parent-child retrieval: index smaller child chunks for accurate matching, then return the surrounding parent section when more context is needed. This often preserves meaning better than making every chunk large. Tables, source code, contracts, and operating procedures may also need specialized parsing. Forcing every content type through the same prose-oriented splitter usually produces fragments that look searchable but cannot support a reliable answer.
Use summaries to compress and retrieval to select
Summaries are useful for preserving a global map of a document: its purpose, major sections, entities, and time range. They are not a safe replacement for source text when wording matters. Compression can remove qualifiers, negation, exceptions, and relationships between values. For contracts, regulations, procedures, or technical specifications, an answer based only on summaries may sound directionally correct while being wrong in a consequential detail. Use summaries for routing and orientation, then verify the answer against retrieved source passages.
Retrieval solves a different problem: choosing a small evidence set from a large corpus. Semantic search handles paraphrases well, while keyword search is often better for product codes, error messages, regulation identifiers, and exact terminology. A hybrid approach is usually practical, followed by reranking or rules that remove weak and duplicate candidates. The query strategy should reflect the question type:
- Focused fact lookup:retrieve a small number of passages containing the exact entity and supporting statement.
- Cross-section comparison:split the question into comparison dimensions, retrieve evidence for each, and synthesize afterward.
- Whole-document summary:summarize sections first, combine those summaries, and retain links to the original passages.
- Conversational follow-up:rewrite the query with necessary references resolved, without carrying the entire chat history forward.
- High-risk answer:require source citations, surface conflicting evidence, and state when the retrieved material is insufficient.
Make the trade-offs observable and testable
There is no universal best chunk size, candidate count, overlap, or summary depth. Evaluate these choices with a representative question set covering exact lookup, multi-passage reasoning, conflicting versions, unanswerable questions, tables, and permission boundaries. Do not score only whether the final answer sounds correct. Check whether the cited text actually supports each conclusion, whether required passages were retrieved, and whether a failure originated in parsing, recall, ranking, or generation. Without that separation, teams often tune the model when the real problem is a damaged document structure or a missed retrieval result.
In production, record token allocation, retrieved chunk identifiers, ranking decisions, citations, and latency while avoiding unnecessary storage of sensitive source text. When evidence exceeds the budget, remove duplicates first, then trim by relevance, authority, and version. If the task genuinely requires broad reasoning, use query decomposition, hierarchical summaries, or staged generation rather than silently dropping passages. The goal is not to send the most text to the model. It is to ensure that every piece of context has a defined purpose and that the engineering team can explain both why an answer is supported and when the system should decline to answer.