Skip to content

Stop giving a verdict about a length nobody measured - #69

Merged
peopleworks merged 1 commit into
mainfrom
verdict-length-floor
Aug 24, 2026
Merged

Stop giving a verdict about a length nobody measured#69
peopleworks merged 1 commit into
mainfrom
verdict-length-floor

Conversation

@peopleworks

Copy link
Copy Markdown
Owner

Closes #59.

The defect

The boundary this build ships was fitted on 90 texts whose shortest is 662 words and whose median
is 2,772. It was being applied, unchanged and unlabelled, to a paragraph somebody pasted into a
textarea — which is most of how the tool is actually used.

That is not a caveat. It is the same defect as #36: a bound measured on one population spent on
another.
VerdictBands already refuses a verdict in a language the corpus never contained, for
exactly that reason. Length is such a population, and now it is treated as one.

What it looks like

A 66-word passage that scores 94/100:

  • Before: Signs of AI writing, in red.
  • After: No verdict at this length, in grey — with all 23 signals, the categories, the rhythm
    chart and the statistics still on the page, and one sentence saying why.

The claim is withheld. The evidence is not.

Why the floor is an observation, not a parameter

It is the shortest text the threshold was measured on: computed by the calibrator, published in the
snapshot, read by VerdictBands.Measured(int).

That choice is the whole design. The committee rejected the previous attempt on five counts — row
composition drifting with source mix, selection and certification on the same corpus, an arbitrary
factor-of-two grid, self-contradictory null semantics, and windows cut from the start of long
documents standing in for short writing. Every one of them was a consequence of fitting something.
None of them exist here, because nothing is fitted. No grid, no windows, no subset chosen until a
number came out.

It also means the claim is weaker than it looks, and the page says so: this does not assert the
tool breaks below 662 words. It asserts that nothing that short was measured.

The direction of the effect is measured (Docs/PARAPHRASE.md, section Length): the same
documents flag 0 of 32 whole and 6 of 32 as 400-word excerpts of themselves. That is why the floor
exists — and why there is deliberately no ceiling. A thesis longer than the corpus is not at
risk, and silencing it for symmetry would withhold a verdict for a reason nobody has evidence for.

Colour is part of the verdict

A withheld verdict used to fall through to the emphasis every surface paints green, so the page
would have refused in words and certified in the loudest channel it has. VerdictEmphasis.Unmeasured
exists for that, and the report, the interface, the folder scan, the rewrite panel and the CLI all
read it.

Four states, and the reasons are separate sentences

"Below the threshold" is a reading — the tool looked and found little. "Shorter than anything
measured" is a refusal to give one. Telling a reader the first when the second is true hands them
a reassurance nobody offered. Both wordings ship in English and Spanish.

The published rate did not move

Same corpus fingerprint 123fa5b9ebca3f29, same 0 of 90 at 25/100, same 4.1% upper bound. Only
the range it covers is now written down — the page and the embedded snapshot both carry
662 – 9,328 words (median 2,772).

One number changed from my first pass and it is worth recording: the floor is 662, not 712. A
naive whitespace split disagrees with the analyzer's own word counter by about 7%, and deriving the
floor with one counter while comparing against the other would have silently rescaled the gate — the
same trap GenreGate documents for its rates.

Two existing tests failed, and both were right to

Their fixtures are paragraphs — as is nearly every fixture in the suite, because that is how people
use the tool
— so they had quietly become tests of the length gate rather than of what they were
written to guard. Fixtures.LongEnough says so out loud, and the finding is the point: the fixtures
were short for the same reason the defect mattered.

Verified

  • 417 tests, 10 new (VerdictLengthTests), including that withholding the verdict does not withhold
    the evidence, that every way of asking agrees, and that the Spanish reader is told the same thing.
  • Rendered and read in the browser in both languages; CLI and Markdown report checked by hand.
  • The citation cross-check does not route through VerdictBands — checked, as the brief asked, so
    the 0/100 citation demo is unaffected.
  • The teacher package needs no change: its rule is that a score is never the reason for a decision,
    which this strengthens.

Not fixed by this

A 900-word essay is above the floor and still shorter than most of the corpus. The floor is a
coverage gate, not a correction. Making the boundary length-dependent needs short complete pre-2022
texts — #66 — and Docs/Calibration/README.md now names that as the most wanted contribution.

🤖 Generated with Claude Code

https://claude.ai/code/session_015PEbbiYSNPw7jE3LrPNhyF

The boundary this build ships was fitted on 90 texts whose shortest is 662
words and whose median is 2,772. It was being applied, unchanged and unlabelled,
to a paragraph somebody pasted into a textarea — which is most of how the tool
is actually used.

That is not a caveat, it is the same defect as #36: a bound measured on one
population spent on another. VerdictBands already refuses a verdict in a
language the corpus never contained, for exactly that reason. Length is such a
population, and now it is treated as one.

The floor is an observation, not a parameter. It is the shortest text the
threshold was measured on, computed by the calibrator and published in the
snapshot, and that choice is the whole design: no grid of lengths, no windows
sliced out of longer documents, no subset picked until a number came out. The
earlier attempt at this was rejected by the committee on five counts and every
one of them was a consequence of fitting something — row composition drifting
with source mix, selection and certification on the same corpus, an arbitrary
factor-of-two grid. None of those exist here because nothing is fitted.

It also means the honest claim is weaker than it looks, and the page says so:
this does not assert the tool breaks below 662 words. It asserts that nothing
that short was measured. The direction of the effect *is* measured, in
Docs/PARAPHRASE.md — the same documents flag 0 of 32 whole and 6 of 32 as
400-word excerpts of themselves — which is why the floor exists and why there is
deliberately no ceiling: a thesis longer than the corpus is not at risk, and
silencing it for symmetry would withhold a verdict for a reason nobody has
evidence for.

What is withheld is the claim, not the evidence. A 66-word passage that scores
94/100 still lists all 23 of its signals, its categories, its rhythm chart and
its statistics. What it no longer does is say "Signs of AI writing" — and what
it no longer does in colour, either. A withheld verdict used to fall through to
the emphasis every surface paints green, so the page would have refused in words
and certified in the loudest channel it has. VerdictEmphasis.Unmeasured exists
for that, and the report, the interface, the folder scan, the rewrite panel and
the CLI all read it.

Four states now, not two, and the reasons are separate sentences: "below the
threshold" is a reading, "shorter than anything measured" is a refusal to give
one, and telling a reader the first when the second is true hands them a
reassurance nobody offered.

The published rate did not move: same corpus fingerprint, same 0 of 90 at
25/100, same 4.1% upper bound. Only the range it covers is now written down.

Two of the existing tests failed on this, and both were right to. Their fixtures
are paragraphs — as is nearly every fixture in the suite, because that is how
people use the tool — so they had quietly become tests of the length gate.
Fixtures.LongEnough says so out loud.

Lowering the floor means measuring short writing somebody composed at that
length, which is #66, and Docs/Calibration/README.md now names it as the most
wanted contribution.

417 tests, 10 new.

Closes #59.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_015PEbbiYSNPw7jE3LrPNhyF
@peopleworks
peopleworks merged commit e115ab1 into main Aug 24, 2026
4 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

The verdict boundary has no length condition, and short passages cross it

1 participant