Monitor the complete service, not just the model API
An enterprise AI application is rarely a single model call. A typical request passes through authentication, prompt assembly, knowledge retrieval, model generation, tool execution, and an external channel such as LINE, an ERP, or a CRM. The model can return successfully while the system still fails because retrieval selected an obsolete document, a tool call was rejected, or the final message never reached the user. Monitoring should therefore follow an entire task from request intake to confirmed outcome.
Infrastructure signals still matter: request volume, success rate, HTTP errors, timeouts, retries, queue depth, CPU, memory, and database connections. Averages, however, often hide a small group of severely delayed requests. Track latency distributions such as p50, p95, and p99, then break total latency into retrieval time, time to first token, generation time, tool execution, and channel delivery. This decomposition shows whether the bottleneck is the model provider, vector database, internal API, or downstream SaaS.
- Availability: end-to-end task success, dependency success, timeouts, and cancelled requests.
- Performance: time to first token, total response time, retrieval latency, tool latency, and queue wait.
- Reliability: retry count, fallback-model activation, duplicate execution, and partially completed tasks.
- Capacity: concurrent requests, rate-limit pressure, connection pools, queue depth, and model quota headroom.
Connect quality metrics to real tasks
AI failures do not always produce exceptions. A response may be fluent yet incomplete, unsupported, or inconsistent with a business rule. Error-code monitoring alone will miss these incidents. Start by separating traffic into task types, such as knowledge questions, summaries, field extraction, support routing, and tool-based actions. Define a testable completion condition for each type. Structured output can be checked against a schema; knowledge answers can be checked for valid citations and source support; agents can be evaluated on tool choice, parameters, authorization, and confirmed final state.
Useful online quality signals include repeated questions, user reformulation, negative feedback, human escalation, abandoned conversations, and reversed actions. They need context: a repeated question may indicate stale source data rather than weak generation, while escalation may be an intentional policy. Retain privacy-safe traces that can be sampled for review, and run a stable offline evaluation set regularly. Segment results by language, task, source, model version, and deployment environment. A global average can conceal a serious regression in a smaller but high-risk workflow.
- Task completion: did the answer resolve the request, or did the requested action actually finish?
- Correctness and grounding: are important claims supported by approved sources, and do citations open to the relevant content?
- Format and policy: does the result satisfy its schema, required fields, language, privacy rules, and business constraints?
- Safety: were prompt injection, unauthorized tool access, sensitive-data exposure, and unsafe content detected or blocked?
Observe RAG, agents, and integrations separately
In a RAG system, an incorrect answer is often a data-pipeline failure rather than a generation failure. Monitor document ingestion, parsing errors, abnormal chunk counts, embedding backlog, index freshness, permission synchronization, and whether retrieval returns the intended sources. Online traces can retain the query, rewritten query, document identifiers, versions, relevance signals, and citation mapping. Avoid placing sensitive source text in general-purpose logs. When quality drops after a knowledge update, this evidence helps isolate the document, chunking strategy, embedding model, or reranker.
For agents and integrations, capture the state of every step instead of recording only the final message. Watch for incorrect tool selection, parameter-validation errors, authorization denials, downstream rate limits, duplicate writes, compensation actions, and time awaiting human approval. Any action with side effects should use an idempotency key or an equivalent safeguard. Record “the model requested an action” separately from “the target system confirmed completion.” An assistant saying that a CRM record was updated is not proof that the CRM accepted the change.
Use a consistent correlation identifier to connect the frontend request, retrieval operation, model call, tool execution, and external response into one trace. Logs explain what happened, metrics expose changes in trends, and traces show where time was spent and where failure began. Without all three, incident diagnosis tends to become guesswork across separate systems.
Make cost and alerts support operational decisions
AI cost includes more than input and output tokens. Embeddings, reranking, vector storage, speech processing, log retention, network transfer, and human review can all be material. Attribute usage by feature, environment, model, tenant, and task type, then track cost per successful task rather than only the total bill. A request may become more expensive because conversation history keeps growing, retrieval adds excessive context, an agent loops through tools, or failures trigger repeated calls. Each cause needs a different correction; switching to a cheaper model is not a complete cost strategy.
Alert thresholds should reflect user impact and service objectives. An isolated dependency failure may deserve logging and an automatic retry. Sustained deterioration in end-to-end success, tail latency, safety behavior, or index freshness may require an operator. Combine fixed thresholds with deviation from an established baseline, and require enough traffic before paging so low-volume noise does not dominate. Every alert should include affected tasks, recent deployments, model and prompt versions, relevant traces, and a runbook with a concrete first action.
Finally, treat models, prompts, evaluation sets, indexes, and integration workflows as versioned production components. Run offline evaluations before a change, release through staged traffic with a rollback path, and compare quality, latency, errors, and cost per successful task after deployment. Mature AI operations is not a dashboard filled with charts. It is a monitoring system in which every important metric helps answer three questions: what users are experiencing, which layer caused it, and what the engineering team should do next.