Replay experiment PoC: local server + replay-case annotation (stock-watchlist-agent)

[ebrevan]

Throwaway PoC for the Replay experiments feature — pairs with DataDog/web-ui#328867 (the "Replay trace" button). Two changes under test-apps/stock-watchlist-agent:

• replay_poc.py — a local Flask server (http://localhost:8787) the Replay button calls. It turns a production trace's case into a one-record LLM Obs Dataset and runs an LLM Obs Experiment whose task re-runs the current local agent code, scored by the app's evaluators, and returns the experiment URL.
• src/agents/orchestrator.py — records the replay case (metadata.replay_input + metadata.replay_output) on the root span on every run, so any production trace is replayable.

Run the local server (to test the Replay button)

cd test-apps/stock-watchlist-agent
python3.12 -m venv .venv
.venv/bin/pip install -e . && .venv/bin/pip install flask

# .env (gitignored) — fill in real values:
#   OPENAI_API_KEY=...      # the agent's provider key
#   DD_API_KEY=...          # trace ingest (32 chars)
#   DD_APP_KEY=...          # Application key (~40 chars) — required for the datasets/experiments API
#   DD_SITE=datadoghq.com   # match your org

.venv/bin/python3.12 replay_poc.py       # serves http://localhost:8787
curl -s http://localhost:8787/health     # -> {"ok":true,"agent":"stock-watchlist-agent"}

Notes:

• Use .venv/bin/python3.12 explicitly — repo python may resolve to an older version.

Once instrumented, every future trace will be replayable

Produce a replayable trace

python -m src.main AAPL GOOGL NVDA     # one run → a trace carrying replay_input/replay_output

After this you can continue with the steps on the FE PR.