File tree Expand file tree Collapse file tree
internal-packages/dashboard-agent Expand file tree Collapse file tree Original file line number Diff line number Diff line change @@ -479,5 +479,7 @@ left standing ages into its own baseline and starts reading as normal.
479479- Invent numbers or claim something doesn't exist beyond a truncated page.
480480- Trust a report whose telemetry is stale.
481481
482- Feedback → #dashboard-agent-feedback, or just tell the agent — every
483- conversation is evaluated and capability gaps are collected automatically.
482+ Feedback → #dashboard-agent-feedback, or just tell the agent — a sample of turns
483+ is scored automatically and capability gaps are collected from it (a tenth by
484+ default, none for a turn that read source, none for an org that opted out; see
485+ [ README.md] ( ./README.md#turn-evals ) ).
Original file line number Diff line number Diff line change @@ -43,3 +43,23 @@ import type { dashboardAgent } from "@internal/dashboard-agent";
4343```
4444
4545Never a value import (see ` src/index.ts ` ).
46+
47+ ## Turn evals
48+
49+ A sampled fraction of turns is scored by an LLM judge (` dashboard-agent-eval-turn ` ), which
50+ writes one ` chat_turn_evals ` row. The rules live in one file, ` src/eval-policy.ts ` :
51+
52+ - ** Sampling.** ` DASHBOARD_AGENT_EVAL_SAMPLE_RATE ` , default ** 0.1** — the judge is a full
53+ model call per turn and nothing reads the rows yet. Golden / CI runs are a separate lane:
54+ ` DASHBOARD_AGENT_EVAL_SAMPLE_RATE_CI ` (default 1) applies only when
55+ ` DASHBOARD_AGENT_EVAL_CONTEXT=ci ` , so neither lane can change the other's rate.
56+ - ** Redaction.** Run payloads and outputs, query result rows, file contents and span
57+ attributes are replaced by their shape before the turn leaves the agent. The row keeps the
58+ judge's derived verdict only — never the question, the answer, or any tool data.
59+ - ** Code mode.** A turn that called a source tool is not judged at all.
60+ - ** Opt-out.** Per-org, via the ` dashboardAgentTurnEvalsEnabled ` feature flag. The agent asks
61+ the API before every judged turn and judges only on an explicit yes.
62+ - ** Retention.** Rows are dropped after 30 days by the webapp's dashboard-agent sweep.
63+
64+ When a document and the code disagree about any of the above, the code is the fact and the
65+ document is the bug: fix the document in the same change.
You can’t perform that action at this time.
0 commit comments