End-to-End Observability in Node.js With OpenTelemetry and Grafana

If your Node.js application is only emitting console.log statements, you are not observing your system — you are guessing about it. When a microservice starts degrading at 2 AM, you need correlated traces, metrics, and logs pointing at the same request, not three disconnected data silos you have to mentally stitch together.

This guide walks through instrumenting a Node.js application with OpenTelemetry and shipping all three observability signals to a self-hosted Grafana stack. No expensive SaaS contracts required.

---

What "Full Observability" Actually Means

The three pillars of observability are not interchangeable:

• Traces show the path of a single request across services and functions, with timing for every span.
• Metrics show the aggregate health of your system over time — request rates, error rates, latency percentiles.
• Logs provide the human-readable narrative of what happened at a specific moment.

Most tutorials pick one. The real value comes when all three are correlated — meaning a spike in your error-rate metric links directly to the trace that caused it, and that trace links to the log line that explains why. OpenTelemetry makes this possible through a shared context propagation model.

---

The Stack

Component	Role
OpenTelemetry SDK (Node.js)	Instruments your app and exports signals
OpenTelemetry Collector	Receives, processes, and routes telemetry
Tempo	Stores and queries distributed traces
Prometheus	Scrapes and stores metrics
Loki	Indexes and stores logs
Grafana	Unified UI across all three backends

This stack runs entirely on your own infrastructure via Docker Compose, making it suitable for staging environments, self-hosted production setups, or regulated environments where data cannot leave your network.

---

Step 1 — Install the OpenTelemetry SDK

Start with the core packages:

npm install @opentelemetry/sdk-node \
  @opentelemetry/auto-instrumentations-node \
  @opentelemetry/exporter-trace-otlp-http \
  @opentelemetry/exporter-metrics-otlp-http \
  @opentelemetry/winston-transport

@opentelemetry/auto-instrumentations-node automatically patches popular libraries — Express, HTTP, pg, Redis, and more — so you get spans without touching your business logic.

---

Step 2 — Bootstrap the SDK Before Your App Loads

Create a dedicated instrumentation.js file and require it before anything else. This ordering is critical — the SDK must patch modules before they are imported.

// instrumentation.js
const { NodeSDK } = require('@opentelemetry/sdk-node');
const { getNodeAutoInstrumentations } = require('@opentelemetry/auto-instrumentations-node');
const { OTLPTraceExporter } = require('@opentelemetry/exporter-trace-otlp-http');
const { OTLPMetricExporter } = require('@opentelemetry/exporter-metrics-otlp-http');
const { PeriodicExportingMetricReader } = require('@opentelemetry/sdk-metrics');

const sdk = new NodeSDK({
  serviceName: 'my-node-api',
  traceExporter: new OTLPTraceExporter({
    url: 'http://otel-collector:4318/v1/traces',
  }),
  metricReader: new PeriodicExportingMetricReader({
    exporter: new OTLPMetricExporter({
      url: 'http://otel-collector:4318/v1/metrics',
    }),
    exportIntervalMillis: 15000,
  }),
  instrumentations: [getNodeAutoInstrumentations()],
});

sdk.start();

Then start your app with:

node -r ./instrumentation.js server.js

---

Step 3 — Correlate Logs With Traces

Logs become dramatically more useful when each log line carries the traceId and spanId of the active request. With Winston, this is straightforward using the OpenTelemetry transport:

const winston = require('winston');
const { OpenTelemetryTransportV3 } = require('@opentelemetry/winston-transport');

const logger = winston.createLogger({
  transports: [
    new winston.transports.Console(),
    new OpenTelemetryTransportV3(), // emits logs as OTLP log records
  ],
  format: winston.format.combine(
    winston.format.timestamp(),
    winston.format.json()
  ),
});

Now every logger.info(...) call inside a traced request will automatically embed the current trace context. In Grafana, you can jump from a slow trace span directly to the log lines generated during that span.

---

Step 4 — Deploy the Collector and Backends

A minimal docker-compose.yml wires everything together. The OpenTelemetry Collector is the central hub — it receives OTLP data from your app and fans it out to the appropriate backend.

Key Collector pipeline configuration (in otel-collector-config.yaml):

• Receivers: otlp on ports 4317 (gRPC) and 4318 (HTTP)
• Processors: batch (improves throughput), memory_limiter (prevents OOM under load)
• Exporters: otlp/tempo for traces, prometheusremotewrite for metrics, loki for logs

Having a Collector between your app and the backends pays off immediately. You can add sampling rules, redact sensitive fields from spans, or swap out a backend — all without redeploying your application.

---

Step 5 — Build Dashboards in Grafana

Once data flows, connect Grafana to all three datasources (Tempo, Prometheus, Loki) and configure datasource links:

1. In your Tempo datasource settings, link to Loki using traceId as the correlation field.
2. In Prometheus, use the service_name label to filter metrics per service.
3. Build a single dashboard combining a Prometheus latency graph, a Tempo trace explorer panel, and a Loki log panel — all filtered by the same service and time range.

The result is a single pane of glass where a latency spike, its causing trace, and the relevant error logs are visible simultaneously without tab-switching.

---

Common Pitfalls to Avoid

• Sampling too aggressively in development: Keep sampling at 100% locally so you catch issues before they reach production.
• Not setting serviceName: Without it, all your telemetry lands under unknown_service, making multi-service environments impossible to navigate.
• Ignoring the Collector's batch processor: Sending every span individually crushes throughput. Batching is not optional at production scale.
• Missing context propagation in async queues: If your app publishes to a message queue, manually inject and extract the W3C traceparent header so traces continue across async boundaries.

---

Why This Matters for Your Project

Shipping a Node.js service without proper observability is like deploying blind. When production incidents happen — and they will — teams that have correlated traces, metrics, and structured logs resolve issues in minutes, not hours. The self-hosted Grafana stack described here costs nothing beyond compute, scales well into the hundreds of thousands of requests per day, and gives your engineering team the same visibility that enterprise SaaS platforms sell at a premium. If you are building or scaling a SaaS product, investing in this infrastructure early is one of the highest-leverage decisions you can make.