Pitfall one: treating RAG as a document upload feature
Many teams start by putting PDFs, policies, manuals, support articles, and internal notes into a vector database, then expect the assistant to answer correctly. Problems appear quickly. Old and new versions are indexed together, tables lose structure, duplicated documents compete with each other, and unofficial notes get retrieved beside approved policy. The model then turns inconsistent context into a confident answer.
The first engineering task in enterprise RAG is knowledge governance, not embedding. Teams need to define which documents are authoritative, who owns them, how versions expire, and what happens after a source is updated. If a document has no clear title, date, owner, product scope, region, or access label, the retrieval layer has little reliable information to work with.
A safer rollout starts with a focused knowledge domain where success and failure are easy to inspect. Internal IT support, product specification lookup, customer service procedures, and sales enablement are common starting points because the boundaries are visible. Once the team has a working pattern for source control, answer format, and feedback, expanding to broader knowledge areas becomes much less fragile.
Pitfall two: tuning the model while neglecting retrieval
RAG quality depends heavily on retrieval. If the system retrieves the wrong passages, misses the important paragraph, or mixes conflicting sources, a better model can only produce a more polished mistake. Common issues include chunks that are too large, chunks that are too small, vector-only search that misses exact terms, no weighting for authoritative documents, and no filtering by date, language, region, or document type.
Retrieval design should follow the use case. A customer service assistant needs standard answers and traceable sources. An engineering knowledge assistant may need to preserve code blocks, error messages, and logs. Contract or compliance search requires precise references and strict version control. These different cases call for different chunking rules, metadata, hybrid search, reranking, and fallback behavior.
- Chunking: Split by headings, sections, tables, and semantic boundaries, not only by fixed character length.
- Metadata: Store source, owner, version, date, product line, language, sensitivity, and permissions so retrieval can filter deliberately.
- Hybrid search: Combine vector search with keyword search for model numbers, error codes, product names, and exact clauses.
- Reranking: Reorder initially retrieved passages to reduce noise from documents that sound similar but do not apply.
- Citations: Return source names or passages so users can verify where the answer came from.
Pitfall three: ignoring permissions, boundaries, and auditability
Enterprise RAG is not just a company search box with a chat interface. It often combines information from cloud drives, ERP, CRM, ticketing systems, project documents, data platforms, and internal databases. If the assistant does not follow the same access rules as the original systems, it can expose customer data, compensation details, contract terms, roadmap material, or engineering documents to the wrong audience.
Permissions need to be part of retrieval, not only a front-end check. User identity, department, role, project membership, data classification, and channel context should shape what the system is allowed to retrieve. This matters even more when the assistant is connected to LINE, Slack, Teams, CRM, or ERP, because each channel may have a different trust boundary.
Auditability is another area teams often postpone. A production RAG system should record who asked a question, which documents were retrieved, what answer was generated, and whether the user accepted or reported the response. These logs are useful for compliance, but they are also the main source for improving retrieval quality, source documents, and prompt behavior.
Pitfall four: approving the system without an evaluation set
RAG demos are easy to make look good. Testers often ask questions that closely match newly indexed documents, and the expected answers are still fresh in everyone’s mind. Real users behave differently. They use shorthand, typos, incomplete context, local terminology, and questions that cross department boundaries. Without an evaluation set, the team cannot tell whether a regression came from document updates, embedding changes, chunking, reranking, or prompt changes.
An evaluation set does not need to be large at the beginning, but it must be realistic. Good sources include support tickets, internal helpdesk questions, sales enablement questions, training material, and historical search queries. Each test item should define the expected answer, acceptable source documents, conditions where the assistant should refuse or say it does not know, and cases that require escalation to a human or a source system.
Approval should check more than writing quality. The team should inspect whether the correct documents were retrieved, whether the answer stayed grounded in those documents, whether uncertainty was handled properly, whether citations are traceable, and whether different roles receive answers that match their permissions. A fluent answer is not the same thing as a reliable enterprise answer.
Pitfall five: launching without an operating model
RAG is not finished on launch day. Enterprise knowledge changes constantly as products evolve, policies are updated, people move teams, and source systems change fields or workflows. Without ownership, feedback loops, and recurring evaluation, the assistant slowly becomes another internal tool that employees stop trusting.
A durable RAG system needs product operations. Teams should review frequent questions, unanswered questions, weak citations, low-confidence answers, and negative feedback. Document updates should trigger reindexing. Major policy, pricing, or process changes need a priority path. Sensitive documents need a fast removal process when access rules change.
The main decision is not which model is newest. It is whether the organization can connect knowledge, systems, permissions, and user workflows in a maintainable way. When ERP, CRM, LINE channels, IoT data platforms, and cloud environments are involved, bringing engineering, data governance, and integration planning together early avoids expensive cleanup later.