Start with an end-to-end deadline
Do not begin with the default timeout in an HTTP library or cloud SDK. Start with the total time the user or upstream service can tolerate, then reserve time for the gateway, application logic, serialization, network transit, and response assembly. What remains is the dependency budget. If one request calls an ERP, a CRM, and a vector database in sequence, each dependency cannot receive the full end-to-end allowance. Otherwise, the innermost timeout may fire only after the caller has already abandoned the request.
Separate connection timeout from response timeout. The connection limit covers DNS resolution, TCP and TLS setup, and waiting for a pooled connection. The response limit controls how long to wait after sending the request, potentially distinguishing time to first byte from time to complete the body. Streaming AI responses also need an inactivity limit after streaming begins. Propagate deadlines and cancellation signals through every service; canceled work that continues in the background still consumes threads, sockets, database connections, and model capacity.
Retry only failures that are transient and safe
Retries are appropriate for short network interruptions, connection resets, explicit retryable responses, or temporary loss of a small portion of a service fleet. They are not a remedy for invalid input, authorization failures, missing resources, or sustained overload. Classify errors using protocol and operation semantics instead of retrying every unsuccessful status. A dependency that is merely slow may become unavailable if every waiting caller adds more work.
Every attempt spends both remaining time and downstream capacity. Configure a maximum attempt count together with exponential backoff, randomized jitter, and a global retry budget. Before another attempt, verify that enough deadline remains to complete it. If not, return a controlled error or fallback immediately. Avoid retries at multiple layers: a gateway, application service, client library, and job runner retrying independently can multiply one logical request into a burst. Put retry ownership in the layer that best understands the operation.
- Prove idempotency: Reads are usually safe, while order creation, payments, and notifications need an idempotency key or a reliable way to reconcile the first attempt.
- Honor dependency signals: Use server-provided delay or throttling guidance when available instead of immediately sending another request.
- Cap aggregate retry traffic: A shared token or retry budget prevents failed work from displacing healthy first attempts.
- Preserve request semantics: Record individual attempts for diagnosis, but report the final logical outcome so monitoring does not count one request as several incidents.
Make circuit breakers sensitive to slowness
A slow dependency can keep returning successful responses while exhausting connection pools, worker queues, and threads. A useful circuit breaker therefore considers timeouts, slow calls, and capacity rejections as well as explicit failures. A measurement window that is too short reacts to minor noise; one that is too long protects the system too late. Define the minimum sample volume, observation window, slow-call threshold, open duration, and number of probes allowed while half-open.
Breaker scope matters as much as its thresholds. One breaker for every ERP operation may let a heavy reporting query block a lightweight inventory lookup. A breaker per endpoint or customer, however, may produce weak samples and an unmanageable configuration. Group calls by dependency, operation behavior, and resource pool, then pair the breaker with concurrency limits or bulkhead isolation. Half-open probes must be few and controlled; releasing every queued request at once can overload a recovering service and reopen the circuit immediately.
Tune from evidence and design the fallback first
Average latency is not enough. Observe tail latency, queue time, connection-pool wait, duration by attempt, final request outcome, breaker transitions, and capacity rejections. Logs and traces should share a request identifier and distinguish initial attempts, retries, and half-open probes. Without that separation, a dashboard may show that the dependency is slow while hiding whether time is actually being spent in the network, server processing, local queueing, or retry backoff.
Decide the failure behavior before tuning thresholds: serve cached data, postpone synchronization, enqueue work, disable a nonessential feature, or return a fast and explicit error. Roll configuration changes out gradually, and exercise them with injected latency, partial failures, and complete outages in a controlled environment. Use this operational checklist:
- Verify that deadlines cross service boundaries and cancellation releases resources.
- Verify that only recoverable, safely repeatable operations are retried.
- Give normal traffic, retries, and half-open probes explicit capacity limits.
- Ensure that opening a circuit produces a predictable fallback instead of overloading another dependency.
- Make alerts distinguish dependency slowness, local capacity exhaustion, and configuration mistakes.
The goal is not to discover one permanent set of numbers. It is to maintain rules that can be derived again from service objectives, observed latency distributions, and available capacity. For integrated API, cloud, and enterprise systems, those rules also need consistent ownership across team and platform boundaries.