What is distributed tracing?
Distributed tracing follows a single request as it moves through services, queues and databases, recording each step as a timed span that shares one trace ID. The result is a tree that shows where the time went and which service failed. Trace context travels between services in headers, usually the W3C traceparent header.
Why distributed tracing matters
In a system of several services, a slow response can come from any hop. Logs from each service show fragments; a trace shows the whole path with timings, so the slow database call three services away is visible in one view.
Traces also reveal structure you did not know about: repeated calls, calls that could run in parallel but do not, and retries that double the load on a struggling dependency.
How to measure distributed tracing
trace duration = end of the root span − start of the root span share of a span = span duration ÷ trace duration self time of a span = its duration − time covered by its child spans
Worked example
A checkout request takes 480 ms at the gateway. Inside it, an auth call takes 25 ms, and the orders service span lasts 410 ms. The orders span contains a database query of 310 ms and a cache read of 4 ms, run one after the other.
- The database query is 310 ÷ 480 = 64.6% of the whole request.
- Self time of the orders service = 410 − (310 + 4) = 96 ms spent in its own code.
- The auth call (25 ms) is small; the query is where optimisation pays.
Result: Without the trace you would know the request takes 480 ms. With it you know that almost two thirds of that is one query, and which one.
Common mistakes
- Broken context propagation. If one service does not forward the traceparent header, the trace splits in two and the path is lost. Check every hop, including queues.
- Random sampling only. Random head sampling keeps 1 in N traces whether or not they are interesting. Keep slow and failed traces deliberately.
- Too many tiny spans. A span per function call makes traces hard to read and expensive to store. Span the boundaries: requests, queries, outbound calls, queue messages.
- Skewed clocks. Spans from hosts with drifting clocks can appear to start before their parent. Keep NTP running everywhere.
How Vigil handles distributed tracing
- Vigil's SDKs and any OpenTelemetry SDK send traces; Vigil keeps the slowest trace per action in each 10-minute window with its full span timeline, so the evidence for a slow request is stored rather than sampled away.
- The trace page flags N+1 patterns (8 or more queries of the same shape in one trace), slow queries of 1 second or more, and chatty requests with 25 or more calls.
- Trace IDs link from log lines and errors, and Time Detective shows traces next to alerts and deploys.
Distributed tracing: frequently asked questions
What is the traceparent header?
It is the W3C Trace Context header that carries the trace ID, the parent span ID and sampling flags from one service to the next, so spans from different services join into one trace.
Do I need distributed tracing for a monolith?
Tracing still helps in a monolith: one request's spans show its database queries, cache calls and outbound HTTP calls in order. The distributed part matters once requests cross services.
How long should traces be kept?
Most teams keep detailed traces for days to weeks and aggregated latency for longer. Keep slow and failed traces longest, because those are the ones you investigate.