A single HTTP request hits your API gateway. By the time it returns a response, it has touched an auth service, a product catalog, a pricing engine, and a third-party tax API. Something in that chain is slow — but your logs show nothing unusual, and your metrics dashboards are perfectly green. This is the microservices debugging trap, and distributed tracing is the only reliable way out.
Why Logs and Metrics Are Not Enough
Logs tell you what happened on a single service at a single point in time. Metrics tell you aggregate behavior over a time window. Neither tool gives you a causal, end-to-end picture of a single request's journey through your system. That picture is exactly what a trace provides: a structured record of every operation a request triggered, across every service, with precise timing for each step.
OpenTelemetry (OTel) is now the industry standard for capturing that picture. It is vendor-neutral, has first-class Node.js support, and integrates cleanly with backends like Grafana Tempo, Jaeger, and Honeycomb.
Core Concepts Before You Touch Code
- Trace: The full journey of a single request, made up of one or more spans.
- Span: A named, timed operation within a trace (e.g., "query users table", "call pricing-service").
- Context propagation: The mechanism that carries trace identifiers across service boundaries, typically via HTTP headers (
traceparent). - Exporter: The component that ships finished spans to your backend (Tempo, Jaeger, etc.).
Setting Up the OpenTelemetry SDK in Node.js
Install the required packages:
npm install @opentelemetry/sdk-node \
@opentelemetry/auto-instrumentations-node \
@opentelemetry/exporter-trace-otlp-http \
@opentelemetry/resources \
@opentelemetry/semantic-conventions
Create a tracing.js file that must be loaded before any other application code:
// tracing.js
const { NodeSDK } = require('@opentelemetry/sdk-node');
const { OTLPTraceExporter } = require('@opentelemetry/exporter-trace-otlp-http');
const { getNodeAutoInstrumentations } = require('@opentelemetry/auto-instrumentations-node');
const { Resource } = require('@opentelemetry/resources');
const { SemanticResourceAttributes } = require('@opentelemetry/semantic-conventions');
const exporter = new OTLPTraceExporter({
url: process.env.OTEL_EXPORTER_OTLP_ENDPOINT || 'http://localhost:4318/v1/traces',
});
const sdk = new NodeSDK({
resource: new Resource({
[SemanticResourceAttributes.SERVICE_NAME]: process.env.SERVICE_NAME || 'my-node-service',
}),
traceExporter: exporter,
instrumentations: [getNodeAutoInstrumentations()],
});
sdk.start();
process.on('SIGTERM', () => sdk.shutdown());
Start your service with:
node -r ./tracing.js server.js
The getNodeAutoInstrumentations() call automatically patches Express, HTTP, gRPC, database clients (pg, mysql, mongoose), and more — without changing a single line of your business logic.
Adding Custom Spans for Business Logic
Auto-instrumentation captures infrastructure-level spans, but your business logic is a black box to it. For anything meaningful — pricing calculations, third-party API calls, cache lookups — add manual spans:
const { trace } = require('@opentelemetry/api');
async function applyDiscount(cart) {
const tracer = trace.getTracer('pricing-service');
return tracer.startActiveSpan('applyDiscount', async (span) => {
try {
span.setAttribute('cart.item_count', cart.items.length);
span.setAttribute('cart.currency', cart.currency);
const result = await computeDiscount(cart);
span.setAttribute('discount.applied', result.discountPercent);
return result;
} catch (err) {
span.recordException(err);
span.setStatus({ code: SpanStatusCode.ERROR });
throw err;
} finally {
span.end();
}
});
}
This gives you a named span with business-relevant attributes that appear directly in the trace timeline.
Propagating Context Across Services
If Service A calls Service B over HTTP, the trace context must travel in the request headers. When using getNodeAutoInstrumentations(), this is handled automatically for http and fetch calls — the SDK injects the traceparent header on outbound requests and extracts it on inbound ones.
For gRPC or message queues (Kafka, RabbitMQ), use the OTel propagator API explicitly:
const { propagation, context } = require('@opentelemetry/api');
// Inject before publishing a message
const carrier = {};
propagation.inject(context.active(), carrier);
message.headers = carrier;
// Extract on the consumer side
const ctx = propagation.extract(context.active(), message.headers);
tracer.startActiveSpan('process-order-event', { context: ctx }, (span) => { ... });
Getting this right is the difference between a fragmented collection of disconnected traces and a single coherent flame graph.
Shipping Traces to Grafana Tempo
Grafana Tempo is an excellent self-hosted choice: it is horizontally scalable, stores traces as object storage, and integrates natively with Grafana dashboards.
A minimal Docker Compose setup:
services:
tempo:
image: grafana/tempo:latest
command: ["-config.file=/etc/tempo.yaml"]
volumes:
- ./tempo.yaml:/etc/tempo.yaml
ports:
- "4317:4317" # OTLP gRPC
- "4318:4318" # OTLP HTTP
grafana:
image: grafana/grafana:latest
ports:
- "3000:3000"
environment:
- GF_AUTH_ANONYMOUS_ENABLED=true
In Grafana, add Tempo as a data source (URL: http://tempo:3200) and enable the Trace to Logs correlation if you are also shipping logs to Loki. Now every trace links directly to the relevant log lines.
Reading the Flame Graph to Find the Bottleneck
Once traces appear in Grafana, open a slow request. The flame graph renders each span as a horizontal bar. Width represents duration. Nesting represents the call hierarchy.
Look for:
- Wide spans with no children: These are leaf operations doing real work — usually database queries or external API calls. A suspiciously wide DB span means a missing index or an N+1 query.
- Sequential spans where parallel is possible: If your service calls three downstream APIs one after the other but none depends on the others, you have a serial bottleneck that async parallelism can eliminate.
- Gaps between spans: Dead time where no span is active often points to connection pool exhaustion or event loop blocking.
- Recurring thin spans at high volume: A loop calling a service 50 times in a single request is an architectural problem no amount of caching will fully solve.
In practice, the first time most teams instrument a mature microservices system, they find at least one N+1 database query and one serialized fan-out that nobody knew existed. The flame graph makes both obvious in under five minutes.
What to Instrument First
Do not try to instrument everything at once. A pragmatic rollout order:
- API Gateway / BFF layer — captures full request scope immediately.
- Highest-traffic internal services — highest return on investment.
- All database and cache clients — auto-instrumentation covers most of these for free.
- Async workers and queue consumers — requires manual context propagation.
- Third-party integrations — wrap external SDK calls in custom spans.
Keeping Overhead Low in Production
OTel's SDK adds minimal CPU overhead, but span volume can grow quickly in high-throughput services. Use tail-based sampling via the OpenTelemetry Collector: keep 100% of error traces and slow traces (e.g., p99 latency), and sample only 5-10% of healthy fast traces. This preserves diagnostic value while controlling storage costs.
Why This Matters for Your Project
Whether you are running three microservices or thirty, uninstrumented systems accumulate latency debt silently. By the time a performance issue becomes visible in user-facing metrics, it has often been compounding for weeks. Distributed tracing with OpenTelemetry gives your engineering team surgical visibility — not just into what broke, but into why and where, down to the millisecond. If you are building or scaling a Node.js-based product, adding OTel instrumentation early is one of the highest-leverage investments you can make in your platform's long-term reliability.
