Skip to content

Local Development

Jump to bottom
Ivan P edited this page Jun 8, 2026 · 1 revision

Local Development — Observability Stack

Developers can run a full observability backend locally using Grafana's grafana/otel-lgtm Docker image before any OpenShift infrastructure is deployed. This lets you instrument a service, verify traces/metrics/logs are emitted correctly, and iterate on instrumentation without needing access to a cluster.

The image bundles the same backends used in the sovereign stack:

  • Grafana — visualization UI
  • Loki — log aggregation
  • Tempo — distributed tracing
  • Mimir — metrics storage
  • Pyroscope — continuous profiling
  • OpenTelemetry Collector — receives OTLP data and fans it out to each backend

Prerequisites

  • Docker + Docker Compose installed locally
  • The service you are instrumenting running locally (bare process or existing docker-compose.yml)

docker-compose Setup

Add the LGTM service to the project's docker-compose.yml (or a dedicated docker-compose.observability.yml):

services:
  lgtm:
    image: grafana/otel-lgtm
    pull_policy: always
    ports:
      - "3000:3000"   # Grafana UI
      - "4317:4317"   # OTel Collector — gRPC (OTLP/gRPC)
      - "4318:4318"   # OTel Collector — HTTP (OTLP/HTTP protobuf)
    environment:
      - GF_AUTH_ANONYMOUS_ENABLED=true
      - GF_AUTH_ANONYMOUS_ORG_ROLE=Admin

GF_AUTH_ANONYMOUS_ENABLED skips the login screen for local dev — remove it if you prefer the default admin/admin login.

Services running inside docker-compose reference the collector at http://lgtm:4317 or http://lgtm:4318. Services running as bare processes on the host use http://localhost:4317 or http://localhost:4318.

Configuring Your Service

Set these environment variables in your service's docker-compose entry (or in your shell for bare-process runs):

environment:
  - OTEL_EXPORTER_OTLP_ENDPOINT=http://lgtm:4318
  - OTEL_EXPORTER_OTLP_PROTOCOL=http/protobuf
  - OTEL_SERVICE_NAME=my-service-name
  - OTEL_LOGS_EXPORTER=otlp        # local dev only — not in Helm values
  - OTEL_METRICS_EXPORTER=otlp
  - OTEL_TRACES_EXPORTER=otlp

For bare-process runs (service not in docker-compose):

export OTEL_EXPORTER_OTLP_ENDPOINT=http://localhost:4318
export OTEL_EXPORTER_OTLP_PROTOCOL=http/protobuf
export OTEL_SERVICE_NAME=my-service-name
export OTEL_LOGS_EXPORTER=otlp     # local dev only — not in Helm values
export OTEL_METRICS_EXPORTER=otlp
export OTEL_TRACES_EXPORTER=otlp

OTEL_LOGS_EXPORTER=otlp is local dev only. In OpenShift, logs are not exported via OTLP — they go to stdout and are scraped by Alloy. See Log Correlation below.

Stack-Specific Notes

Python (aiohttp, FastAPI)

Use opentelemetry-instrument for zero-code auto-instrumentation:

pip install opentelemetry-distro opentelemetry-exporter-otlp
opentelemetry-bootstrap -a install
opentelemetry-instrument python app.py

The SDK picks up OTEL_* env vars automatically. No code changes required to get basic traces, metrics, and logs.

Node.js / TypeScript

Use the @opentelemetry/auto-instrumentations-node package:

npm install @opentelemetry/auto-instrumentations-node @opentelemetry/sdk-node @opentelemetry/exporter-trace-otlp-http

Launch with:

node --require @opentelemetry/auto-instrumentations-node/register app.js

Or set NODE_OPTIONS=--require @opentelemetry/auto-instrumentations-node/register in the docker-compose environment block.

Log Correlation

In OpenShift, logs are not sent via OTLP. They go to stdout and are scraped by Alloy, which ships them to Loki. Trace correlation works because the OTel SDK injects trace_id and span_id directly into the log records before they reach stdout — Alloy just transports them.

Locally, there is no Alloy to scrape stdout. Instead, OTEL_LOGS_EXPORTER=otlp tells the SDK to push logs directly to the LGTM OTel Collector via OTLP, which forwards them to Loki. The SDK still attaches trace context to each log record, so trace_id is populated in Loki the same way.

The end result in Grafana is identical: logs in Loki with trace_id that links to spans in Tempo. The transport is different, the correlation is not.

Local (LGTM) OpenShift
Log transport OTLP push → OTel Collector → Loki stdout → Alloy scraping → Loki
OTEL_LOGS_EXPORTER otlp not set
trace_id in logs injected by OTel SDK into OTLP log records injected by OTel SDK into stdout log output
Correlation in Grafana ✓ ✓

Python: OTEL_PYTHON_LOG_CORRELATION=true injects trace_id and span_id into Python's standard logging output. Set this in both local dev and Helm values — it controls the injection, not the transport.

Node.js / TypeScript: @opentelemetry/instrumentation-pino or @opentelemetry/instrumentation-winston (included in auto-instrumentations-node) injects trace context into structured log records. As with Python, this is independent of whether logs are sent via OTLP or stdout.

What to validate locally

In Grafana Explore, open a log line in Loki and confirm trace_id is present as a field. That confirms the SDK is injecting trace context correctly — the same injection that Alloy will pick up in OpenShift. If trace_id is missing from local logs, it will also be missing in production.

Verifying It Works

  1. Start the stack: docker compose up lgtm
  2. Start your service with the OTEL_* env vars set
  3. Make a few requests through your service
  4. Open Grafana at http://localhost:3000
  5. Navigate to Explore and check:
    • Tempo — should see traces with spans for incoming HTTP requests
    • Loki — should see log lines correlated with trace IDs (if structured logging is in place)
    • Mimir / Prometheus — should see HTTP request duration metrics

A visible trace in Tempo is the go/no-go signal that the instrumentation, OTel Collector, and SDK are all wired up correctly.

Effort Estimate

Task Effort
Add LGTM service to each project's docker-compose ~½h per repo
Document SDK setup and env vars per stack (Python, Node.js) ~1–2h total
Validate each service emits traces end-to-end locally included in Phase 2 per-service effort

Documentation and docker-compose changes can be done incrementally as each service is onboarded in Phase 2. The LGTM setup should be added to each service's repo as part of its onboarding issue.

Differences from OpenShift

The instrumentation code is identical between local and deployed environments. Two env vars differ:

Variable Local dev OpenShift (Helm values)
OTEL_EXPORTER_OTLP_ENDPOINT http://lgtm:4318 Alloy collector endpoint in the namespace
OTEL_LOGS_EXPORTER otlp not set — logs go via stdout → Alloy

Everything else (OTEL_TRACES_EXPORTER, OTEL_METRICS_EXPORTER, OTEL_SERVICE_NAME, OTEL_PYTHON_LOG_CORRELATION, etc.) is the same in both environments.

Clone this wiki locally