Start with the question you need to answer. Use metrics to establish how much of a service is affected, traces to follow the work inside selected requests, and logs to inspect specific events. Correlate the signals through a stable service identity, a common time range, and request or trace identifiers where appropriate.
Collecting three kinds of data is not enough. If a dashboard shows an error spike but you cannot connect it to a request, a dependency, or a relevant event, the missing connection becomes part of the incident.
The OpenTelemetry observability primer describes metrics as numerical aggregation, logs as event records, and traces as connected spans that follow operations. The practical choice is which evidence can narrow the current uncertainty fastest, not which signal is universally best.
One checkout failure, three views
We ran a small Python fixture containing six sequential synthetic checkout requests. Each request performed an instrumented inventory lookup. Two lookups deliberately waited 20 milliseconds; the other four waited 1 millisecond. One of the slow requests then emitted an inventory-unavailable event. Those waits are injected conditions, not measurements of a real inventory system.
The fixture used Python's monotonic performance counter for elapsed time, emitted parent and child span records, and aggregated request outcomes. It used only the standard library: there was no OpenTelemetry SDK, collector, distributed service, or telemetry backend. This keeps the evidence inspectable while limiting what the example can demonstrate.
Its aggregate outcome was:
{"requests": {"ok": 5, "error": 1}}That tells us one of six attempts failed. It does not tell us which operation caused the failure or whether the failure represents normal traffic. A six-request fixture is too small and too artificial to justify a production error budget or percentile target.
The event record carried the missing identity:
{
"trace_id": "00000000000000000000000000000005",
"event": "inventory_unavailable",
"route": "/checkout",
"error_code": "synthetic_timeout"
}Filtering the spans by that trace identifier returned exactly two records: checkout and its child inventory.lookup. Their measured durations showed that the deliberate wait occurred inside the lookup. The event explained the selected outcome. The aggregate counted its impact. These records corroborate the fixture's programmed behavior; they do not establish a production root cause.
Move from impact to a testable explanation
For a real incident, begin with the user-visible symptom. “Checkout errors increased after a release” is more useful than “CPU looks high” because it establishes what must recover. Compare the error ratio and request volume over the same interval. Ten errors among ten requests has a different meaning from ten among a million.
Next, segment by a small set of meaningful dimensions: service, environment, route template, outcome class, and deployment version if your retention model supports it. A single failing route may warrant a different investigation from errors across every route. Check whether the apparent increase came from a traffic change, an instrumentation change, or a different denominator before treating it as a regression.
Select traces that represent the symptom. Inspect parent-child relationships and critical-path timing. A long checkout span with a short inventory child suggests missing work elsewhere, such as queueing before that child or an uninstrumented dependency. In contrast, a long inventory child narrows the search to that call and its surroundings; it still does not prove the database itself was slow.
Finally, inspect relevant events. A dependency timeout, a validation rejection, and a connection-pool wait can all produce similar user-visible failures but require different corrective actions. Use event codes and structured fields to separate them, then test the explanation against additional requests and a control interval.
Choose dimensions you can afford to keep
Metrics work well when they aggregate events into stable groups. A unique request identifier as a metric attribute turns one stream into a growing collection of nearly unique series. Keep detailed identifiers in traces or appropriately protected logs, and keep metric dimensions bounded by the question the metric answers.
OpenTelemetry's metrics documentation distinguishes instruments such as counters and histograms. For the checkout example, a counter can accumulate attempts by outcome; a duration histogram can retain a distribution suitable for later aggregation. A log line for each duration is a different storage and query design, even when both originate from the same operation.
Do not attach raw customer names, authorization headers, or full request bodies merely to make correlation convenient. A request identifier can connect records without copying the request's sensitive contents into every telemetry system. Review who can query those identifiers, how long they remain useful, and whether they expose information when joined with other data.
Understand what an empty search cannot prove
Missing trace data does not prove a request never happened. Sampling, dropped exports, collector failures, retention expiry, and mismatched time ranges can all leave gaps. Similarly, no error log does not prove success: the failure may occur before the logger executes, or the logger may use a different event code.
Record the coverage behind an investigation. Were all requests counted? Which traces were retained? Did every service propagate context? Are clocks comparable across hosts? Use monotonic clocks for local elapsed durations, but do not compare their raw values across machines as if they share one timeline.
An operationally useful conclusion states both evidence and limits: “The error ratio increased for checkout, sampled failing traces spent their longest child interval in inventory lookup, and corresponding events reported dependency timeouts.” That justifies investigating the inventory path. “The database is the root cause” needs further evidence.
Begin improving observability at the first missing connection in that chain. Add the event, identity, or span that would resolve a recurring ambiguity, then exercise a controlled failure and confirm that an investigator can find it. The six-request fixture demonstrates that correlation step; your own service needs a representative test of its actual instrumentation and delivery path.
