docs(testing-agents): the sample-size cost of statistical significance

[AshwinUgale]

Follow-up to fullsend-ai/experiments#39, per @rh-hemartin's request on that PR: capture the "a large number of cases is needed for statistical significance" idea in the problem docs.

Adds two small things to docs/problems/testing-agents.md:

• A paragraph in the Non-determinism section on the sample-size cost of statistical significance — why a point estimate at small n doesn't support the claim made on it (19/20 doesn't certify a 90% floor), that the required trials grow as the claim approaches the agent's true rate, and two practical consequences (gate on the lower confidence bound; size the suite to the smallest regression worth catching).
• A partial-resolution note on open question docs: Add agent-compatible code problem document #1, pointing to the experiment for the sizing math. The policy half — what pass rate is acceptable — is left open, since that's a risk decision rather than a statistic.

Worked numbers and the utility live in the merged experiment: 0026-eval-statistical-significance.

[qodo-code-review]

PR Summary by Qodo

Docs: explain sample-size cost of statistical significance in agent testing

📝 Documentation 🕐 Less than 10 minutes

AI Description

• Explain why small-n pass rates don’t imply strong performance floors in non-deterministic agent
 tests.
• Add practical guidance: gate on lower confidence bounds and size suites to the smallest regression
 worth catching.
• Update an open question with a partial resolution and link to Experiment 0026 for sizing math.

Diagram

graph TD
  D["docs/problems/testing-agents.md"] --> N["Non-determinism section"] --> E["Experiment 0026"]
  D --> O["Open question #1"] --> E

Loading

High-Level Assessment

The following are alternative approaches to this PR:

1. Add a small rule-of-thumb table in the doc

• ➕ Makes guidance actionable without leaving the page
• ➕ Helps reviewers pick initial trial counts quickly
• ➖ Risk of over-generalizing across tasks/judges
• ➖ Requires maintaining numbers/assumptions as guidance evolves

2. Create a dedicated “Statistical testing” appendix page

• ➕ Keeps core problem statement concise while housing deeper material
• ➕ Creates a natural place for future expansions (power, CI choices, graded judges)
• ➖ More navigation overhead for readers
• ➖ May be overkill for the current incremental clarification

3. Vendor the sizing utility into this repo

• ➕ Keeps methodology and tooling co-located with the docs
• ➕ Enables consistent usage in CI/eval tooling
• ➖ Adds maintenance surface area and dependency/versioning concerns
• ➖ May duplicate ownership with the experiments repo

Recommendation: The PR’s approach (tight paragraph + link to the worked experiment) is appropriate for a problem doc and keeps the repo lightweight. If readers still struggle to operationalize the guidance, the next best incremental step would be a small rule-of-thumb table or a short appendix page, without importing tooling into this repo.

Files changed (1) +3 / -1

Documentation (1) +3 / -1

testing-agents.mdDocument sample-size requirements for statistical significance in agent evals +3/-1

Document sample-size requirements for statistical significance in agent evals

• Adds a paragraph in the Non-determinism section explaining why small-n point estimates can’t support strong performance claims, and recommends gating on lower confidence bounds and sizing suites to detect meaningful regressions. Updates open question #1 with a partial resolution and a link to Experiment 0026 for the detailed sizing math.

docs/problems/testing-agents.md

[qodo-code-review]

Code Review by Qodo

🐞 Bugs (0) 📘 Rule violations (0) 📜 Skill insights (0)

1. fullsend-ai link in problem doc ✗ Dismissed 📘 Rule violation § Compliance

Description

Core problem docs under docs/problems/ must not include organization-specific identifiers outside
docs/problems/applied/, but the new/modified text links directly to
github.com/fullsend-ai/experiments/.... Move this org-specific reference into an applied doc and
link to it from the core document instead.

Code

docs/problems/testing-agents.md[19]

+The cost of that statistical framing is easy to underestimate: reliable significance requires a large number of trials, and the count grows as the claim gets bolder relative to the agent's true performance. A run that scores 19/20 does not certify a 90% floor — that sample is consistent with a true rate as low as ~76%. Distinguishing a real ten-point regression from run-to-run noise takes on the order of a hundred-plus trials per variant, and small differences in a graded (1–5) judge score can take thousands. Two practical consequences: gate against the *lower bound* of a confidence interval rather than the point estimate, and size the suite to the smallest regression worth catching. [Experiment 0026](https://github.com/fullsend-ai/experiments/tree/main/0026-eval-statistical-significance) works through the sizing math and the numbers.

Relevance

●●● Strong

Team often removes org-specific references from core problem docs; move details to applied doc per
precedent.

PR-#3064
PR-#5646

ⓘ Recommendations generated based on similar findings in past PRs

Evidence

PR Compliance ID 1062038 requires organization-specific identifiers to be placed under
docs/problems/applied/<org-name>/ and not appear inline in core problem docs. The changed lines
add/modify direct links to the fullsend-ai experiments repository in this core problem doc.

Rule 1062038: Separate generic and organization-specific problem documents
docs/problems/testing-agents.md[19-19]
docs/problems/testing-agents.md[373-373]

Agent prompt

The issue below was found during a code review. Follow the provided context and guidance below and implement a solution

## Issue description
The core problem document `docs/problems/testing-agents.md` adds organization-specific references (direct links to the `fullsend-ai/experiments` repo) outside the `docs/problems/applied/` hierarchy.

## Issue Context
Per compliance, organization-specific identifiers/examples must live under `docs/problems/applied/<org-name>/...`, with core problem docs staying generic.

## Fix Focus Areas
- docs/problems/testing-agents.md[19-19]
- docs/problems/testing-agents.md[373-373]

ⓘ Copy this prompt and use it to remediate the issue with your preferred AI generation tools

---

2. CI parameters unspecified ✗ Dismissed 🐞 Bug ≡ Correctness

Description

The new significance paragraph introduces a concrete lower-bound figure (“~76%” for 19/20) and
recommends gating on the “lower bound” of a confidence interval, but does not state the confidence
level or interval convention (e.g., one- vs two-sided / method), making the guidance
non-reproducible and easy to misapply.

Code

docs/problems/testing-agents.md[19]

+The cost of that statistical framing is easy to underestimate: reliable significance requires a large number of trials, and the count grows as the claim gets bolder relative to the agent's true performance. A run that scores 19/20 does not certify a 90% floor — that sample is consistent with a true rate as low as ~76%. Distinguishing a real ten-point regression from run-to-run noise takes on the order of a hundred-plus trials per variant, and small differences in a graded (1–5) judge score can take thousands. Two practical consequences: gate against the *lower bound* of a confidence interval rather than the point estimate, and size the suite to the smallest regression worth catching. [Experiment 0026](https://github.com/fullsend-ai/experiments/tree/main/0026-eval-statistical-significance) works through the sizing math and the numbers.

Relevance

●●● Strong

They tend to tighten docs to make numeric claims verifiable; specifying CI/confidence parameters is
a straightforward clarification.

PR-#5269

ⓘ Recommendations generated based on similar findings in past PRs

Evidence

The doc asserts a specific implied floor (~76%) from a 19/20 outcome and separately advises gating
on a confidence-interval lower bound, but neither location defines the confidence level or interval
convention, which is required to interpret or reproduce a lower bound.

docs/problems/testing-agents.md[15-21]
docs/problems/testing-agents.md[371-374]

Agent prompt

The issue below was found during a code review. Follow the provided context and guidance below and implement a solution

### Issue description
The doc gives a specific implied floor (~76% from 19/20) and recommends gating on a “lower bound”, but it doesn’t define the confidence level or interval convention, so readers can’t reproduce the number or implement consistent pass/fail gating.

### Issue Context
The linked experiment may contain the full math, but the doc text itself reads like a self-contained rule of thumb. Without stating (at least) the confidence level and whether the bound is one-sided, teams can interpret “lower bound” differently and end up with inconsistent thresholds.

### Fix Focus Areas
- docs/problems/testing-agents.md[19-20]
- docs/problems/testing-agents.md[371-374]

ⓘ Copy this prompt and use it to remediate the issue with your preferred AI generation tools

---

Context used

✅ Compliance rules (platform): 54 rules

✅ Skills: writing-user-docs, writing-adrs

_{To customize comments, go to the Qodo configuration screen, or learn more in the docs.}

[waynesun09]

Both qodo-code-review findings addressed inline: the experiments link is this project's own sibling repo (not a downstream-consumer org-specific detail requiring applied/), and the confidence-interval specifics are intentionally left to the linked experiment per this repo's problem-doc style (present trade-offs, don't prescribe). LGTM.

[github-actions]

E2E tests did not run

E2E tests run automatically for org/repo members and collaborators on pull requests.

For other contributors, a maintainer must add the ok-to-test label after the latest push.

See E2E testing guide for details.

[github-actions]

Site preview

Preview: https://4c62bad8-site.fullsend-ai.workers.dev

Commit: 9a86f594bed67709f362baf5acda21f5c8e5fb14

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.

📢 Thoughts on this report? Let us know!

[fullsend-ai-retro]

🤖 Finished Retro · ✅ Success · Started 2:31 PM UTC · Completed 2:42 PM UTC

Commit: 9a86f59 · View workflow run →

[fullsend-ai-retro]

Retro: PR #5999 — docs(testing-agents): the sample-size cost of statistical significance

Workflow outcome: Clean, efficient merge. No proposals filed.

PR #5999 is a tiny human-authored docs-only change (+3/−1, single file docs/problems/testing-agents.md) by AshwinUgale, adding a paragraph on sample-size costs of statistical significance and a partial-resolution note on an open question. No fullsend agents (code, review, fix, triage) were involved — this is expected for a human-authored PR on a non-agent branch (docs/testing-agents-significance).

Timeline

1. 2026-08-06 21:53 — PR opened by AshwinUgale.
2. 2026-08-06 21:54–21:59 — qodo-code-review[bot] posted a PR summary and 2 inline findings.
3. 2026-08-07 13:57–13:58 — waynesun09 dismissed both qodo findings with clear rationale and approved the PR.
4. 2026-08-07 14:15 — Branch synchronized (merge from main).
5. 2026-08-07 14:28 — PR merged by waynesun09.
6. 2026-08-07 14:29 — Retro dispatched on merge (run 31187806927).

Qodo false positives (both dismissed)

• Compliance rule 1062038 (org-specific link in core problem doc): False positive. The link points to fullsend-ai/experiments, the project's own sibling repo, not a downstream consumer's org-specific detail. The same file already contains multiple links to fullsend-ai/* repos on other lines. The human reviewer correctly dismissed this.
• Confidence level unspecified (~76% lower-bound figure without stating confidence level): Technically valid observation, but contextually inappropriate. Problem docs in this repo present trade-offs and point to detailed resources — they don't prescribe exact parameters. The specifics live in the linked Experiment 0026. The human reviewer correctly dismissed this per the repo's problem-doc conventions.

Why no proposals

Every improvement opportunity identified is already covered by existing open issues:

• Retro dispatch on zero-agent PRs: This retro run itself is the main inefficiency — no fullsend agents were involved in this PR, so the retro has limited analytical value. This is extensively tracked: #5943 (skip retro for any unmerged PR with zero agent involvement), #5399 (skip for merged PRs with zero meaningful agent involvement), and #5817 (consolidate 22+ overlapping skip-retro issues). This run provides additional evidence: a merged, human-authored, docs-only PR with only third-party bot activity triggered a full retro dispatch.
• Qodo false positive reduction: Covered by #4107 (overlapping findings between qodo and fullsend review agent) and fullsend-ai/agents#150 (suppress low-value compliance findings on trivial PRs).
• Docs-only PR review quality: Covered by #5298 (promote docs sub-agent budget for docs-only PRs) and #2678 (deprioritize findings on planning/design documents).

Filing new proposals would contribute to the exact duplication problem that #5817 is trying to solve.