Start with operational questions, not blanket instrumentation
A tracing rollout often begins by installing an SDK in every service. That can produce thousands of spans without explaining why a user waited. Start by listing the questions the system must answer: Where did a request queue? Which external dependency was slow? Was a message retried? How long did it wait before a consumer began processing it? Which synchronous and asynchronous components participated in one business operation? Those questions define useful span boundaries and attributes.
For HTTP or gRPC, the usual parent-child model works well. An entry service creates or continues a trace, injects its context into outbound calls, and each downstream service creates a child span. Database operations, external APIs, and substantial processing stages can have their own spans. Avoid tracing every helper function. A span is most valuable when the operation has a distinct failure mode, resource wait, or responsible team. Excessively fine instrumentation increases telemetry cost and makes the critical path harder to see.
Model queue causality instead of pretending it is HTTP
Queues change the shape of a trace. A producer may publish a message that is processed much later, delivered more than once, consumed in a batch, or distributed to several subscribers. Treat publishing, waiting in the broker, receiving, and processing as separate concerns. Inject standard trace context into message headers, then extract it in the consumer. Record the messaging system, topic or queue, operation, message identifier, and redelivery status using stable semantic conventions where available.
A consumer span can be a child of the producer span when there is one producer, one consumer, and a short message lifetime. Span links are usually a better representation for fan-out, delayed jobs, and batch processing. They preserve causality without creating a single trace that appears open for hours. When one batch combines multiple messages, do not arbitrarily choose one message as the parent. Create a batch-processing span, link it to the source contexts, and add per-message spans only when independent diagnosis is worth the volume.
- Propagation: Prefer W3C Trace Context and verify that gateways, frameworks, broker clients, and job runners preserve rather than rewrite its headers.
- Queue delay: Capture the publish or enqueue time so dashboards can distinguish time waiting for capacity from time executing consumer code.
- Retries: Keep the original business identifier, but create a new processing span for every attempt and record attempt number, redelivery, and failure reason.
- Data safety: Never attach full payloads, tokens, customer records, or raw query parameters. Use controlled categories, hashes, or internal identifiers when correlation is necessary.
Design sampling and attributes for investigations
Collecting every trace is convenient in development, but production volume, storage cost, and query performance usually require sampling. A fixed head-sampling rate can discard the rare error or unusually slow request that matters most. A practical design combines affordable head sampling at ingress with tail sampling in the collector or backend, retaining traces with errors, timeouts, high latency, or critical operations. Confirm that the sampling decision propagates consistently; otherwise, traces will appear to lose services in the middle of a request.
Attribute design matters more than span count. Establish low-cardinality fields shared across teams, such as service, environment, operation, dependency, queue, message type, result, and retry reason. Tenant, order, or device identifiers may help investigations, but they require explicit decisions about cardinality, privacy, and retention. Store them only in controlled telemetry when appropriate. Define error semantics as well: a valid business rejection is not necessarily a system failure, while a caught exception followed by successful compensation should not automatically mark the root operation as failed. The final status should reflect what the caller observed, with events preserving intermediate problems.
Use a repeatable workflow to find the critical path
When latency occurs, begin at the entry span and identify the end-to-end duration and critical path. Compare application execution, downstream calls, and queue waiting. If the API finishes quickly but the message starts late, inspect consumer lag, concurrency limits, partition assignment, and downstream throttling. If pickup is immediate but processing is slow, examine database locks, external dependencies, connection pools, and retry behavior. Repeated spans may indicate application retries, broker redelivery, or successful work followed by a failed acknowledgement; the remedies are very different.
Before production rollout, validate complete scenarios deliberately: a normal request, downstream timeout, delayed delivery, consumer failure and retry, dead-letter handling, and fan-out. Do more than confirm that a trace exists. Check that service names are stable, timestamps make sense, errors appear on the correct spans, and an alert or structured log can link directly to the same trace. Tracing should then work alongside metrics and logs: metrics show the scope and trend of a problem, traces explain individual execution paths, and logs provide detailed evidence. With those responsibilities clear, distributed tracing becomes a routine engineering tool rather than a dashboard opened only during an incident.