Start with the decision, not the model
Many projects begin with a request for an enterprise assistant, a RAG system, or an AI agent. Those labels describe possible solutions, not an implementable requirement. The first engineering question is what a person must decide or do differently: respond to a customer, locate an order, compare specifications, classify a request, or investigate an operational anomaly. If the output does not lead to a defined decision, workflow step, or system action, the team cannot determine what data is required or how success should be measured.
Ask users to walk through the current process using real screens and representative inputs. Identify where the task begins, which systems they consult, which judgment they apply, and where they record the outcome. Then ask whether a workflow change, better search, or deterministic rule could solve the same problem. AI is justified when its flexibility with language or ambiguous inputs creates enough value to accept added latency, cost, and variability.
- Trigger: Who uses the capability, and what event starts the work? Is it a manual request, a schedule, or an event from LINE, an ERP, or a CRM?
- Resulting action: Does the system offer advice, draft content, create a ticket, update a record, send a message, or execute a transaction?
- Failure cost: What happens when the output is wrong, incomplete, or late? Which failure is least tolerable?
- Human control: Which outputs require approval, and what evidence does the reviewer need before acting?
- Baseline: Must the solution outperform a manual process, existing search, or a rules engine? Define the comparison before development.
Turn “we have data” into testable conditions
Available data is not automatically usable data. Discovery must establish format, quality, ownership, permissions, and update frequency. Documents may be distributed across file servers, Google Drive, SharePoint, databases, and personal folders, with several versions of the same policy. A retrieval system can find a passage perfectly and still produce the wrong answer if the source is obsolete. Ask who can declare a source authoritative, who updates it, and whether old material must be removed, isolated, or marked with an effective period.
Inspect representative files instead of relying on an inventory. Can scanned PDFs be read accurately? Do tables, images, or attachments contain essential facts? Are Chinese and English terms used consistently? Can document permissions be enforced for each user? When records include personal information, confidential business data, or customer content, decide what may be sent to an external model, what must be masked, and what may appear in logs.
- Source of truth: Which systems are authoritative, and which source wins when records conflict?
- Freshness: How soon must the AI see an update? Is real-time synchronization necessary, or is a scheduled batch sufficient?
- Access control: Must the AI preserve department, role, customer, and project-level permissions from the source system?
- Traceability: Must answers cite a source and version? What happens to conversations and audit records after a source is deleted?
- Evaluation data: Can domain owners provide representative questions, expected answers, acceptable variations, and cases the system must refuse?
Map integration boundaries and failure modes
The hardest part of an enterprise AI system is often identity, synchronization, existing APIs, and exception handling rather than the model itself. Draw the entire data flow from the user interface through the model, knowledge layer, and enterprise applications. For every connection, confirm whether an API exists, who owns it, whether a test environment is available, and what rate limits apply. Network boundaries, firewalls, VPNs, and single sign-on requirements may determine where components can run. If an ERP exposes inventory only through nightly exports, the design must not promise real-time availability.
Separate read, recommend, and execute permissions. Looking up an order has a different risk profile from modifying it; drafting a LINE response is different from sending it automatically. For each dependency, ask what should happen when it times out, returns incomplete data, rejects a write, or becomes unavailable. A production design needs explicit fallback behavior, such as keyword search, a retry queue, human escalation, or a hard stop on high-risk actions.
- Latency and capacity: How long will users wait, what peak workload is expected, and what model-cost ceiling must the design respect?
- Consistency: If caches, vector indexes, and source systems disagree, should the system warn, block the action, or tolerate a short delay?
- Observability: Will logs capture prompt versions, retrieved content, tool calls, model output, human edits, and the final business action?
- Recovery: Can failed operations be retried safely, reversed, or transferred to a person without duplicate writes?
Define executable acceptance and release criteria
“The answers should be accurate” is not an acceptance criterion. Each task needs its own test dimensions. Knowledge assistants should be evaluated for source selection, faithfulness to the source, and appropriate refusal when evidence is missing. Extraction workflows should be checked for required fields and formats. Agentic workflows also require verification of tool selection, parameters, permissions, and the resulting state in downstream systems. Gather normal, boundary, unauthorized, and failure scenarios, then have the people who own the actual process confirm the expected outcome.
Release criteria should include response time, availability, budget, security review, data retention, and support procedures. Determine which tests must run again after a model, prompt, knowledge source, or integration API changes. Decide who can approve a release, how versions are recorded, and how the team will roll back when quality declines. Assign separate owners for business rules, knowledge content, and technical operations; otherwise outdated content or a changed workflow will often be misdiagnosed as a model problem.
Good discovery should end with defined use cases, a data inventory, an integration diagram, a risk register, evaluation scenarios, and a phased scope. The first release will usually be safer when it limits sources, users, and executable actions, then expands automation after reviewing real usage and failure patterns. An integration team adds the most value here by turning uncertain requirements into boundaries that can be estimated, tested, secured, and maintained—not by promising a particular model before the system is understood.