feat: add OTel observability benchmark demo

[Ladas]

Summary

• Add extraPortMappings support to ClusterSpec for macOS KIND port forwarding
• Add full OTel observability benchmark demo with composable stacks
• Both scenarios with A/B/C benchmarks (baseline vs OTel noop vs OTel full export)

Stack deployed by forge up

Prometheus + Grafana 11.x, Tempo, Loki + Promtail, OTel Collector, MLflow, Fortio echo, inference-sim, Praxis proxy (OTel-enabled)

Quick start

See examples/otel-benchmark/README.md for full instructions.

Dashboards

Dashboard	Description
Proxy Overview	Request rate, latency P50/P99, requests by method
OTel Traces	Searchable trace table with clickable Trace IDs
Benchmark Results	CPU/memory for praxis + collector, RPS under load
AI/LLM Golden Signals	P95 latency, throughput, future AI token metrics
Structured Logs	Loki log volume + praxis access log entries

Benchmark Results (2026-08-14)

Scenario 1: Core Praxis OTel Overhead

Platform: KIND (1 node, macOS arm64) | Backend: Fortio echo | Rate: 2000 RPS x 30s x 3 runs
10 spans per request (root + 8 per-filter + upstream_exchange).

Config	P50 (us)	P99 (us)	RPS	P50 delta	P99 delta
Baseline (praxis:dev)	389	617	2000	--	--
OTel noop (praxis:dev-otel)	383	670	1999	-1.5%	+8.6%
OTel full (praxis:dev-otel + export)	422	820	2000	+8.5%	+32.9%

Scenario 2: AI Praxis with Inference Sim

Platform: KIND (1 node, macOS arm64) | Backend: inference-sim (mock LLM) | Rate: 500 RPS x 30s x 3 runs
11 spans per request (core spans + response_body phase).

Config	P50 (us)	P99 (us)	RPS	P50 delta	P99 delta
AI Baseline (praxis-ai:dev)	728	1528	500	--	--
AI OTel noop (praxis-ai:dev-otel)	592	8903	500	-18.7%	+482.7%
AI OTel full (praxis-ai:dev-otel + export)	528	1541	500	-27.5%	+0.9%

Note: AI OTel images are built from a different code branch with performance improvements in the OTel code path, which explains the P50 decrease. The noop P99 spike reflects first-run warmup.

AI Span Tree

POST /v1/chat/completions -> inference-sim  (root)
  |-- filter:request_id:request
  |-- filter:access_log:request
  |-- filter:router:request        -> routes /v1/* to inference cluster
  |-- filter:load_balancer:request
  |-- filter:load_balancer:response
  |-- filter:router:response
  |-- filter:access_log:response
  |-- filter:request_id:response
  |-- filter:access_log:response_body
  +-- upstream_exchange [inference-sim:8000]

Dependencies (unmerged PRs)

Build requires cherry-picking these OTel PRs onto praxis main:

PR	Description
praxis#956	TCP lifecycle spans
praxis#957	trace_context filter
praxis#958	upstream spans
praxis#959	span events
praxis#960	access log trace ID
praxis#970	semconv followup
praxis#982	Tokio runtime fix for OTel batch exporter
praxis#983	per-filter child spans
ai#751	OTel tracing export fix