Stop giving a verdict about a length nobody measured - #69
Merged
Conversation
The boundary this build ships was fitted on 90 texts whose shortest is 662 words and whose median is 2,772. It was being applied, unchanged and unlabelled, to a paragraph somebody pasted into a textarea — which is most of how the tool is actually used. That is not a caveat, it is the same defect as #36: a bound measured on one population spent on another. VerdictBands already refuses a verdict in a language the corpus never contained, for exactly that reason. Length is such a population, and now it is treated as one. The floor is an observation, not a parameter. It is the shortest text the threshold was measured on, computed by the calibrator and published in the snapshot, and that choice is the whole design: no grid of lengths, no windows sliced out of longer documents, no subset picked until a number came out. The earlier attempt at this was rejected by the committee on five counts and every one of them was a consequence of fitting something — row composition drifting with source mix, selection and certification on the same corpus, an arbitrary factor-of-two grid. None of those exist here because nothing is fitted. It also means the honest claim is weaker than it looks, and the page says so: this does not assert the tool breaks below 662 words. It asserts that nothing that short was measured. The direction of the effect *is* measured, in Docs/PARAPHRASE.md — the same documents flag 0 of 32 whole and 6 of 32 as 400-word excerpts of themselves — which is why the floor exists and why there is deliberately no ceiling: a thesis longer than the corpus is not at risk, and silencing it for symmetry would withhold a verdict for a reason nobody has evidence for. What is withheld is the claim, not the evidence. A 66-word passage that scores 94/100 still lists all 23 of its signals, its categories, its rhythm chart and its statistics. What it no longer does is say "Signs of AI writing" — and what it no longer does in colour, either. A withheld verdict used to fall through to the emphasis every surface paints green, so the page would have refused in words and certified in the loudest channel it has. VerdictEmphasis.Unmeasured exists for that, and the report, the interface, the folder scan, the rewrite panel and the CLI all read it. Four states now, not two, and the reasons are separate sentences: "below the threshold" is a reading, "shorter than anything measured" is a refusal to give one, and telling a reader the first when the second is true hands them a reassurance nobody offered. The published rate did not move: same corpus fingerprint, same 0 of 90 at 25/100, same 4.1% upper bound. Only the range it covers is now written down. Two of the existing tests failed on this, and both were right to. Their fixtures are paragraphs — as is nearly every fixture in the suite, because that is how people use the tool — so they had quietly become tests of the length gate. Fixtures.LongEnough says so out loud. Lowering the floor means measuring short writing somebody composed at that length, which is #66, and Docs/Calibration/README.md now names it as the most wanted contribution. 417 tests, 10 new. Closes #59. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_015PEbbiYSNPw7jE3LrPNhyF
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes #59.
The defect
The boundary this build ships was fitted on 90 texts whose shortest is 662 words and whose median
is 2,772. It was being applied, unchanged and unlabelled, to a paragraph somebody pasted into a
textarea — which is most of how the tool is actually used.
That is not a caveat. It is the same defect as #36: a bound measured on one population spent on
another.
VerdictBandsalready refuses a verdict in a language the corpus never contained, forexactly that reason. Length is such a population, and now it is treated as one.
What it looks like
A 66-word passage that scores 94/100:
chart and the statistics still on the page, and one sentence saying why.
The claim is withheld. The evidence is not.
Why the floor is an observation, not a parameter
It is the shortest text the threshold was measured on: computed by the calibrator, published in the
snapshot, read by
VerdictBands.Measured(int).That choice is the whole design. The committee rejected the previous attempt on five counts — row
composition drifting with source mix, selection and certification on the same corpus, an arbitrary
factor-of-two grid, self-contradictory null semantics, and windows cut from the start of long
documents standing in for short writing. Every one of them was a consequence of fitting something.
None of them exist here, because nothing is fitted. No grid, no windows, no subset chosen until a
number came out.
It also means the claim is weaker than it looks, and the page says so: this does not assert the
tool breaks below 662 words. It asserts that nothing that short was measured.
The direction of the effect is measured (
Docs/PARAPHRASE.md, section Length): the samedocuments flag 0 of 32 whole and 6 of 32 as 400-word excerpts of themselves. That is why the floor
exists — and why there is deliberately no ceiling. A thesis longer than the corpus is not at
risk, and silencing it for symmetry would withhold a verdict for a reason nobody has evidence for.
Colour is part of the verdict
A withheld verdict used to fall through to the emphasis every surface paints green, so the page
would have refused in words and certified in the loudest channel it has.
VerdictEmphasis.Unmeasuredexists for that, and the report, the interface, the folder scan, the rewrite panel and the CLI all
read it.
Four states, and the reasons are separate sentences
"Below the threshold" is a reading — the tool looked and found little. "Shorter than anything
measured" is a refusal to give one. Telling a reader the first when the second is true hands them
a reassurance nobody offered. Both wordings ship in English and Spanish.
The published rate did not move
Same corpus fingerprint
123fa5b9ebca3f29, same 0 of 90 at 25/100, same 4.1% upper bound. Onlythe range it covers is now written down — the page and the embedded snapshot both carry
662 – 9,328 words (median 2,772).One number changed from my first pass and it is worth recording: the floor is 662, not 712. A
naive whitespace split disagrees with the analyzer's own word counter by about 7%, and deriving the
floor with one counter while comparing against the other would have silently rescaled the gate — the
same trap
GenreGatedocuments for its rates.Two existing tests failed, and both were right to
Their fixtures are paragraphs — as is nearly every fixture in the suite, because that is how people
use the tool — so they had quietly become tests of the length gate rather than of what they were
written to guard.
Fixtures.LongEnoughsays so out loud, and the finding is the point: the fixtureswere short for the same reason the defect mattered.
Verified
VerdictLengthTests), including that withholding the verdict does not withholdthe evidence, that every way of asking agrees, and that the Spanish reader is told the same thing.
VerdictBands— checked, as the brief asked, sothe 0/100 citation demo is unaffected.
which this strengthens.
Not fixed by this
A 900-word essay is above the floor and still shorter than most of the corpus. The floor is a
coverage gate, not a correction. Making the boundary length-dependent needs short complete pre-2022
texts — #66 — and
Docs/Calibration/README.mdnow names that as the most wanted contribution.🤖 Generated with Claude Code
https://claude.ai/code/session_015PEbbiYSNPw7jE3LrPNhyF