Skip to content

Commit 99b8933

Browse files
committed
docs(dashboard-agent): make the eval docs match the code
The docs claimed every turn is judged; the sampled rate, the redaction rule, the code-mode skip, the opt-out and the retention period are now written down in one place.
1 parent 263b324 commit 99b8933

2 files changed

Lines changed: 24 additions & 2 deletions

File tree

internal-packages/dashboard-agent/GUIDEBOOK.md

Lines changed: 4 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -479,5 +479,7 @@ left standing ages into its own baseline and starts reading as normal.
479479
- Invent numbers or claim something doesn't exist beyond a truncated page.
480480
- Trust a report whose telemetry is stale.
481481

482-
Feedback → #dashboard-agent-feedback, or just tell the agent — every
483-
conversation is evaluated and capability gaps are collected automatically.
482+
Feedback → #dashboard-agent-feedback, or just tell the agent — a sample of turns
483+
is scored automatically and capability gaps are collected from it (a tenth by
484+
default, none for a turn that read source, none for an org that opted out; see
485+
[README.md](./README.md#turn-evals)).

internal-packages/dashboard-agent/README.md

Lines changed: 20 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -43,3 +43,23 @@ import type { dashboardAgent } from "@internal/dashboard-agent";
4343
```
4444

4545
Never a value import (see `src/index.ts`).
46+
47+
## Turn evals
48+
49+
A sampled fraction of turns is scored by an LLM judge (`dashboard-agent-eval-turn`), which
50+
writes one `chat_turn_evals` row. The rules live in one file, `src/eval-policy.ts`:
51+
52+
- **Sampling.** `DASHBOARD_AGENT_EVAL_SAMPLE_RATE`, default **0.1** — the judge is a full
53+
model call per turn and nothing reads the rows yet. Golden / CI runs are a separate lane:
54+
`DASHBOARD_AGENT_EVAL_SAMPLE_RATE_CI` (default 1) applies only when
55+
`DASHBOARD_AGENT_EVAL_CONTEXT=ci`, so neither lane can change the other's rate.
56+
- **Redaction.** Run payloads and outputs, query result rows, file contents and span
57+
attributes are replaced by their shape before the turn leaves the agent. The row keeps the
58+
judge's derived verdict only — never the question, the answer, or any tool data.
59+
- **Code mode.** A turn that called a source tool is not judged at all.
60+
- **Opt-out.** Per-org, via the `dashboardAgentTurnEvalsEnabled` feature flag. The agent asks
61+
the API before every judged turn and judges only on an explicit yes.
62+
- **Retention.** Rows are dropped after 30 days by the webapp's dashboard-agent sweep.
63+
64+
When a document and the code disagree about any of the above, the code is the fact and the
65+
document is the bug: fix the document in the same change.

0 commit comments

Comments
 (0)