Skip to content

The two-axis review does not ask the question that eleven of PR #461's fifteen defects answered to #462

Description

@orioltf

What this is

PR #461 took seven Copilot rounds, two local /code-review passes, one coherence sweep and one Consumer run to reach green. Roughly fifteen defects surfaced. Every one was real; none reached develop. The process worked, and it worked at a cost worth reducing.

This ticket is about the cost. The defects fall into classes, the classes are not evenly served by the checks this repo runs, and one class dominates.

What the fifteen were, and who caught them

Class Count Found by
Local self-inconsistency — a schema requiring nine keys beside a prompt saying "pass through what you read"; a comment header describing the node above it; a table row contradicting its own second sentence; a count written beside a list that changed length; one instruction stating the same rule three times, two of them disagreeing; a sentence left mangled because an edit landed beside the text it should have replaced 11 Copilot found most. The local /code-review found none of them on its first pass, and found three on a later pass on the same tree, after a third question was added to its prompt
Cross-file drift — five surfaces still describing /qa without a node this PR added; an ADR guaranteeing behaviour written before the floor existed 4 local /code-review, and a hand sweep the maintainer asked for
Behavioural — a prompt wording that would have made a branch unreachable; a $node.output.<field> reference to a field the node is instructed to omit; a verdict printed to a stream that also carried prose, so it never parsed 3 split between both
Spec fidelity caught during grilling, before implementation

Eleven of fifteen are one class, and it is the class the two-axis review does not ask about. Its two axes ask "does this follow documented standards" and "does this match the spec". Neither asks "is this file consistent with itself".

Three of those eleven were introduced by the fix for the round before, which is the second finding here: rounds 2 and 3 and the mangled sentence are one failure mode, not three defects.

Proposals

Ranked by the evidence above. Not pre-decided — the fourth carries a real tension and the grilling should settle it.

1. A third axis on /code-review: self-consistency

.claude/skills/code-review/SKILL.md defines two axes. Add a third that reads each changed file against itself: an output schema against the prompt that fills it, a comment header against the node beneath it, a table row against the behaviour it describes, a stated count against the list beside it, a rule against every other statement of the same rule in the same prompt.

Evidence it works: this axis was added by hand to a re-run of the same review, on the same commit the two-axis version had already passed, and it immediately returned three defects — including a sentence with a duplicated clause and a dangling verb, sitting in a prompt an agent executes at run time.

2. Run the review after the last fix, not only before the first push

Four of Copilot's findings were present in the exact commit the local review read. A one-shot review of a 700-line prose diff under-samples; the review is cheap and the second pass is where the churn shows.

3. A stated rule for how a fix is applied: replace, never append

Rounds 2 and 3 and the mangled step are the same failure — a correction landed beside the text it should have replaced, leaving two statements that disagree, or one sentence carrying half of each. Root AGENTS.md already tells a reviewer to read their diff with the unchanged lines around each edit in view. What is missing is the paragraph-level instruction to the editor: after fixing, re-read the edited paragraph whole, against itself, before moving on. This is the only proposal here that prevents rather than detects.

4. A CI check for dangling $node.output.<field> references — and the tension it carries

A reference to a field a node is instructed to omit is the class that kills an Archon run at template resolution with no useful error. It is mechanically checkable: parse each Box YAML, collect every $<node>.output.<field> token, and confirm the named node's output_format declares that field.

The tension, stated rather than hidden. Root AGENTS.md names "adding a module plus a test so that prose becomes testable" as a defect, and #381 deleted exactly that from unic-archon-dlc. Two things distinguish this from what #381 removed: it would live in the repo's CI and never install into a Consumer, and it validates a reference graph, not prose. It is still code watching prose, and the grilling should decide whether that line holds. If the answer is no, say so and close this proposal — a stated ceiling beats a check nobody trusts.

5. Give Copilot the repo's own conventions

Copilot's review offers .github/skills/code-review/SKILL.md as a way to feed it repo context. Today it reviews with none, which is why several of its rounds re-raised decisions this repo had already settled and recorded. Cheap; may also shorten the rounds.

Not proposed, deliberately

What this did not decide

  • Whether proposal 4 crosses the no-code line, and if it does, whether anything replaces it.
  • Whether the third axis is a new agent or a widening of the Standards axis. A third agent costs a third fan-out; widening risks the same under-sampling that missed eleven defects.
  • Whether proposal 3 belongs in root AGENTS.md, in the code-review skill, or in both.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    needs-specsEnough info from reporter; ready to write a specrepoArea: monorepo-wide / cross-cutting worktech-debtInternal restructuring; no behaviour change

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions