Define the process before choosing the KPI
AI initiatives often begin with a sensible but vague objective: answer customers faster, reduce document work, or help sales teams find information. None of these statements is directly testable. The first task is to draw the process boundary: which event starts the clock, which state ends it, which human approvals remain, and whether queues, rework, escalation, and system delays count. If the before and after measurements use different boundaries, an apparent improvement says little about the AI system.
A baseline is not a target. It is a reproducible description of current performance under stated conditions. The unit of analysis should match the work, such as one ticket, contract, qualified opportunity, or device alert. Department-wide hours are usually a weak denominator because workload, staffing, and seasonality change. Per-unit measures make periods more comparable. When cases vary substantially in difficulty, classify routine, standard, and exceptional work before comparing results. Otherwise, assigning easier cases to the AI can create a misleading success.
The observation window should capture normal variation, but measurement should not wait indefinitely for perfect data. A practical approach is to build an initial baseline from existing system records, then use a limited manual sample to capture quality, corrections, and hidden effort. Every reported value should answer four questions: where the data originated, who or what created it, when it was recorded, and whether the same query can reproduce it.
Measure speed, quality, cost, and adoption together
Speed alone can hide work transferred elsewhere. An AI may draft responses quickly while supervisors spend longer reviewing them. A document assistant may save an operator time but create extra field corrections for a downstream team. Measure the end-to-end process, including handling time, waiting time, human touch time, rework, and escalation. Cost should include more than model calls: integration maintenance, data preparation, monitoring, access control, and the manual fallback required when automation fails.
Quality measures should reflect business risk. For a RAG assistant, test whether citations support the answer, whether the correct document version was used, and whether the system abstains when evidence is missing. For LINE customer service, evaluate intent routing, collection of required information, and handoff to an agent. For ERP or CRM automation, inspect field accuracy, duplicate writes, workflow state, and auditability. Give each metric a measurement card containing at least:
- Definition: the numerator, denominator, start event, end event, and calculation.
- Evidence source: event logs, tickets, ERP, CRM, financial records, or manual review.
- Segmentation: case type, channel, language, risk level, and user role.
- Exception rules: treatment of missing data, reopened cases, batch work, outages, and overrides.
- Decision use: the conditions that trigger expansion, revision, suspension, or manual handling.
Design comparisons that support attribution
A before-and-after comparison is convenient but easily confounded. The launch may coincide with a new knowledge base, revised forms, staff reassignment, or a policy change. Unless those changes are recorded, the team can see the combined outcome but cannot identify the cause. Where feasible, run the AI in shadow mode: let it produce recommendations without controlling the live process, then compare its result with the existing human outcome. If parallel operation is impractical, phase deployment across comparable teams or case types while keeping measurement definitions unchanged.
Comparison groups do not need to resemble a perfect academic experiment, but their differences must be visible. New and experienced employees, routine requests and high-risk cases, or peak and ordinary periods should not be pooled without qualification. Segment results by the most important complexity and risk factors, and show workload and exception volume alongside the outcome. When evidence is limited, label it directional instead of presenting an overly precise return calculation.
Generative AI also needs a stable quality evaluation set. Build it from real categories of work and include common requests, boundary cases, missing evidence, permission constraints, and prompts the system should refuse. Re-run the same core cases after changing the model, prompt, retrieval method, or document chunking. Preserve the input, model and prompt versions, retrieved evidence, output, and human judgment. For consequential outputs, reviewers need consistent scoring rules and a shared failure taxonomy; otherwise, the evaluation will drift with individual preferences.
Set decision gates and ownership before development
The purpose of a baseline is not merely to populate a dashboard. It is to support a decision. Before implementation, document the minimum acceptable quality, risk measures that must not deteriorate, tolerable review effort, and the evidence required to enter the next deployment stage. A gate does not have to be one number. It can be a combined rule: quality remains acceptable, end-to-end time improves, severe errors do not increase, and users rely on the output in the production workflow.
Assign both a business owner and a technical owner to every material metric. The business owner confirms that the definition reflects real operations and risk. The technical owner ensures the underlying events are recorded, queryable, and traceable. A dashboard is only the presentation layer. The essential asset is the data contract covering event names, identifiers, timestamps, versions, override reasons, and final states. When a source system changes, version the baseline and calculation logic so that a silent schema change does not manufacture an improvement.
Finally, define stopping conditions in advance. Pause expansion when missing data prevents reproduction of a core measure, users routinely bypass the system, review effort cancels the expected saving, or risk events cannot be detected reliably. Fix the workflow and instrumentation before scaling the model. A mature AI program does not try to prove that a model is always valuable. It establishes when the system helps, for which work, under what constraints, and when a human must take over. In projects spanning LINE, ERP, CRM, cloud services, and IoT platforms, an integration team should make this measurement capability part of the architecture rather than adding reports after launch.