What this is
PR #461 took seven Copilot rounds, two local /code-review passes, one coherence sweep and one Consumer run to reach green. Roughly fifteen defects surfaced. Every one was real; none reached develop. The process worked, and it worked at a cost worth reducing.
This ticket is about the cost. The defects fall into classes, the classes are not evenly served by the checks this repo runs, and one class dominates.
What the fifteen were, and who caught them
| Class |
Count |
Found by |
| Local self-inconsistency — a schema requiring nine keys beside a prompt saying "pass through what you read"; a comment header describing the node above it; a table row contradicting its own second sentence; a count written beside a list that changed length; one instruction stating the same rule three times, two of them disagreeing; a sentence left mangled because an edit landed beside the text it should have replaced |
11 |
Copilot found most. The local /code-review found none of them on its first pass, and found three on a later pass on the same tree, after a third question was added to its prompt |
Cross-file drift — five surfaces still describing /qa without a node this PR added; an ADR guaranteeing behaviour written before the floor existed |
4 |
local /code-review, and a hand sweep the maintainer asked for |
Behavioural — a prompt wording that would have made a branch unreachable; a $node.output.<field> reference to a field the node is instructed to omit; a verdict printed to a stream that also carried prose, so it never parsed |
3 |
split between both |
| Spec fidelity |
— |
caught during grilling, before implementation |
Eleven of fifteen are one class, and it is the class the two-axis review does not ask about. Its two axes ask "does this follow documented standards" and "does this match the spec". Neither asks "is this file consistent with itself".
Three of those eleven were introduced by the fix for the round before, which is the second finding here: rounds 2 and 3 and the mangled sentence are one failure mode, not three defects.
Proposals
Ranked by the evidence above. Not pre-decided — the fourth carries a real tension and the grilling should settle it.
1. A third axis on /code-review: self-consistency
.claude/skills/code-review/SKILL.md defines two axes. Add a third that reads each changed file against itself: an output schema against the prompt that fills it, a comment header against the node beneath it, a table row against the behaviour it describes, a stated count against the list beside it, a rule against every other statement of the same rule in the same prompt.
Evidence it works: this axis was added by hand to a re-run of the same review, on the same commit the two-axis version had already passed, and it immediately returned three defects — including a sentence with a duplicated clause and a dangling verb, sitting in a prompt an agent executes at run time.
2. Run the review after the last fix, not only before the first push
Four of Copilot's findings were present in the exact commit the local review read. A one-shot review of a 700-line prose diff under-samples; the review is cheap and the second pass is where the churn shows.
3. A stated rule for how a fix is applied: replace, never append
Rounds 2 and 3 and the mangled step are the same failure — a correction landed beside the text it should have replaced, leaving two statements that disagree, or one sentence carrying half of each. Root AGENTS.md already tells a reviewer to read their diff with the unchanged lines around each edit in view. What is missing is the paragraph-level instruction to the editor: after fixing, re-read the edited paragraph whole, against itself, before moving on. This is the only proposal here that prevents rather than detects.
4. A CI check for dangling $node.output.<field> references — and the tension it carries
A reference to a field a node is instructed to omit is the class that kills an Archon run at template resolution with no useful error. It is mechanically checkable: parse each Box YAML, collect every $<node>.output.<field> token, and confirm the named node's output_format declares that field.
The tension, stated rather than hidden. Root AGENTS.md names "adding a module plus a test so that prose becomes testable" as a defect, and #381 deleted exactly that from unic-archon-dlc. Two things distinguish this from what #381 removed: it would live in the repo's CI and never install into a Consumer, and it validates a reference graph, not prose. It is still code watching prose, and the grilling should decide whether that line holds. If the answer is no, say so and close this proposal — a stated ceiling beats a check nobody trusts.
5. Give Copilot the repo's own conventions
Copilot's review offers .github/skills/code-review/SKILL.md as a way to feed it repo context. Today it reviews with none, which is why several of its rounds re-raised decisions this repo had already settled and recorded. Cheap; may also shorten the rounds.
Not proposed, deliberately
What this did not decide
- Whether proposal 4 crosses the no-code line, and if it does, whether anything replaces it.
- Whether the third axis is a new agent or a widening of the Standards axis. A third agent costs a third fan-out; widening risks the same under-sampling that missed eleven defects.
- Whether proposal 3 belongs in root
AGENTS.md, in the code-review skill, or in both.
What this is
PR #461 took seven Copilot rounds, two local
/code-reviewpasses, one coherence sweep and one Consumer run to reach green. Roughly fifteen defects surfaced. Every one was real; none reacheddevelop. The process worked, and it worked at a cost worth reducing.This ticket is about the cost. The defects fall into classes, the classes are not evenly served by the checks this repo runs, and one class dominates.
What the fifteen were, and who caught them
/code-reviewfound none of them on its first pass, and found three on a later pass on the same tree, after a third question was added to its prompt/qawithout a node this PR added; an ADR guaranteeing behaviour written before the floor existed/code-review, and a hand sweep the maintainer asked for$node.output.<field>reference to a field the node is instructed to omit; a verdict printed to a stream that also carried prose, so it never parsedEleven of fifteen are one class, and it is the class the two-axis review does not ask about. Its two axes ask "does this follow documented standards" and "does this match the spec". Neither asks "is this file consistent with itself".
Three of those eleven were introduced by the fix for the round before, which is the second finding here: rounds 2 and 3 and the mangled sentence are one failure mode, not three defects.
Proposals
Ranked by the evidence above. Not pre-decided — the fourth carries a real tension and the grilling should settle it.
1. A third axis on
/code-review: self-consistency.claude/skills/code-review/SKILL.mddefines two axes. Add a third that reads each changed file against itself: an output schema against the prompt that fills it, a comment header against the node beneath it, a table row against the behaviour it describes, a stated count against the list beside it, a rule against every other statement of the same rule in the same prompt.Evidence it works: this axis was added by hand to a re-run of the same review, on the same commit the two-axis version had already passed, and it immediately returned three defects — including a sentence with a duplicated clause and a dangling verb, sitting in a prompt an agent executes at run time.
2. Run the review after the last fix, not only before the first push
Four of Copilot's findings were present in the exact commit the local review read. A one-shot review of a 700-line prose diff under-samples; the review is cheap and the second pass is where the churn shows.
3. A stated rule for how a fix is applied: replace, never append
Rounds 2 and 3 and the mangled step are the same failure — a correction landed beside the text it should have replaced, leaving two statements that disagree, or one sentence carrying half of each. Root
AGENTS.mdalready tells a reviewer to read their diff with the unchanged lines around each edit in view. What is missing is the paragraph-level instruction to the editor: after fixing, re-read the edited paragraph whole, against itself, before moving on. This is the only proposal here that prevents rather than detects.4. A CI check for dangling
$node.output.<field>references — and the tension it carriesA reference to a field a node is instructed to omit is the class that kills an Archon run at template resolution with no useful error. It is mechanically checkable: parse each Box YAML, collect every
$<node>.output.<field>token, and confirm the named node'soutput_formatdeclares that field.The tension, stated rather than hidden. Root
AGENTS.mdnames "adding a module plus a test so that prose becomes testable" as a defect, and #381 deleted exactly that fromunic-archon-dlc. Two things distinguish this from what #381 removed: it would live in the repo's CI and never install into a Consumer, and it validates a reference graph, not prose. It is still code watching prose, and the grilling should decide whether that line holds. If the answer is no, say so and close this proposal — a stated ceiling beats a check nobody trusts.5. Give Copilot the repo's own conventions
Copilot's review offers
.github/skills/code-review/SKILL.mdas a way to feed it repo context. Today it reviews with none, which is why several of its rounds re-raised decisions this repo had already settled and recorded. Cheap; may also shorten the rounds.Not proposed, deliberately
What this did not decide
AGENTS.md, in thecode-reviewskill, or in both.