Classify the failure before switching providers
An HTTP error is only the most visible failure mode. Production systems also encounter connection timeouts, rate limits, interrupted streams, invalid structured output, incorrect tool arguments, unexpectedly strict safety filters, and responses that are technically successful but no longer useful. Treating every symptom as a provider outage can make matters worse: an application bug, bad retrieval result, or invalid prompt may simply be sent to the next provider, multiplying latency and cost without fixing the cause.
The routing layer should separate failures into retryable, failover-eligible, and terminal categories. A transient network error may justify a small, bounded retry. Rate limiting or a regional outage may justify changing deployments or providers. Authentication failures, policy conflicts, invalid requests, and sensitive-data restrictions should normally stop immediately. Timeout budgets should also reflect the workload. An interactive assistant and an offline document-processing job should not share identical retry behavior.
Manage capability contracts, not interchangeable model names
A backup model must satisfy the application’s capability contract. That contract may cover context size, Traditional Chinese quality, structured output, tool calling, image input, safety behavior, data residency, and provider retention settings. Two endpoints that expose a similar API are not necessarily equivalent in semantics or failure behavior. RAG pipelines, ERP updates, and multi-step agents are especially sensitive to small differences in formatting, citations, and tool selection.
Define routing policy by workload instead of maintaining one global primary-and-backup pair. Knowledge answering, document extraction, classification, code generation, and operations with side effects should each have their own candidate models, timeouts, validators, and fallback boundaries.
- Capability fit: Verify required context capacity, structured output, multimodal input, and tool support.
- Risk boundary: Confirm whether data may be sent to that provider or region and how prompts and logs are protected.
- Output compatibility: Test field names, enumerations, citations, and tool arguments against the same validators.
- Cost and capacity: Ensure the backup can absorb traffic, or explicitly reduce context and output limits during degradation.
- Task criticality: A weaker model may be acceptable for low-risk questions, but not for payments, access changes, or record creation.
Build a degradation ladder instead of a binary failover
A useful strategy is a ladder of progressively smaller capabilities. The first step may switch regions, deployments, or comparable models within the same provider, minimizing behavioral differences. The next step can route to another provider through a tested prompt template and output adapter. Further degradation may disable image input, retrieve fewer documents, shorten conversation history, or return a plain-text answer. The final step may serve an approved cached result, provide search links, queue the request for later, or clearly state that the task cannot currently be completed safely.
Every level must preserve business meaning. If an assistant cannot access current ERP data, it should not turn stale information into a confident answer. If a LINE bot cannot confirm an order write, it must not claim success. Separate answering capability from execution capability: answering may move to a smaller model, while execution must use idempotency keys, transaction state, and validation results to decide whether to retry, queue, or stop.
- For read-only questions, switch models while retaining source citations and data timestamps.
- For long-running generation, persist job state and move work to a background queue.
- For side-effecting operations, check transaction status before resubmission to prevent duplicate records or notifications.
- When safety or format guarantees cannot be maintained, disabling the feature is often better than returning an untrustworthy result.
Make failover observable, testable, and reversible
Routing telemetry should capture the provider, model, region, workload, timeout stage, retry count, failover reason, validation result, and final degradation level without placing complete sensitive prompts or documents in logs. HTTP success alone is not enough. Monitor time to first token, total latency, empty responses, schema failures, tool-call errors, missing citations, content filtering, and queue growth. Circuit-breaker decisions should combine these signals rather than depend on a single status code.
A circuit breaker needs a cooldown period, limited probe traffic, and explicit recovery criteria. When a provider appears healthy again, do not immediately return all traffic. Use health checks or shadow requests to verify latency, formatting, and critical task behavior, then restore traffic gradually. Hedged requests can reduce tail latency for suitable read-only work, but they add cost and must never allow two requests to trigger the same external action.
Regular failure exercises expose gaps that architecture documents miss. Test timeouts, throttling, stream interruption, corrupted output, regional loss, simultaneous primary and backup failure, and the return to normal service. Each exercise should verify the user-facing message, alert ownership, queued work, transaction consistency, and post-incident evidence. An integration team can help connect these controls across models, enterprise data, and operational workflows, but the essential outcome is a system whose behavior remains predictable when its preferred model is unavailable.