Skip to content

review-pr: harden Pre-Verdict Audit against comment-quality halo effect - #60

Draft
warp-agent-staging[bot] wants to merge 1 commit into
eval-base/019ff1e1from
eval/harden-pre-verdict-audit-pva
Draft

review-pr: harden Pre-Verdict Audit against comment-quality halo effect#60
warp-agent-staging[bot] wants to merge 1 commit into
eval-base/019ff1e1from
eval/harden-pre-verdict-audit-pva

Conversation

@warp-agent-staging

@warp-agent-staging warp-agent-staging Bot commented Aug 12, 2026

Copy link
Copy Markdown
Contributor

Summary

Hardens the ## Pre-Verdict Audit section of the review-pr skill so a reviewer can no longer let a comment's writing quality or technical accuracy substitute for actually checking it against the target repo's commenting guidelines.

Background: a reviewer using this skill let a comment's writing quality/technical accuracy ("well-written, explains a genuinely hard bug") substitute for actually checking it against the target repo's commenting guidelines, and missed confirmed rule violations as a result — a "comment-quality halo effect."

Changes

  • The Comments bullet in ## Pre-Verdict Audit now requires listing every added/changed comment (doc comment or inline) one by one with its file:line, and checking each individually — rather than a holistic pass — against the repo's commenting guidelines (or, absent explicit guidelines, against the commenting distribution of existing code in the project).
  • Explicitly states that compliance must be evaluated independently of a comment's writing quality, technical accuracy, or how subtle/important the issue it describes is — none of those qualities excuses a violation of an applicable guideline or a clear mismatch with the codebase's own norms.
  • No rule names or categories are hardcoded, so the skill stays generic across repos with or without explicit written commenting guidelines.
  • The Tests bullet, the schema, severity labels, safety rules, evidence rules, suggestion-block constraints, and the diff-line-annotation contract are all unchanged.

Verification

git diff shows exactly one changed hunk in .agents/skills/review-pr/SKILL.md: one line removed, one line added (the Tests bullet is untouched). No other file is touched.

Conversation: https://staging.warp.dev/conversation/92699808-32bb-4839-8a21-b03e5eb1733f
Run: https://oz.staging.warp.dev/runs/019ff4a4-6558-7f41-8b16-9b4805ab6d2b
This PR was generated with Oz.

Require enumerating each added/changed comment individually (with
file:line) when checking it against the repo's commenting guidelines,
and explicitly forbid treating a comment's writing quality, technical
accuracy, or the subtlety/importance of what it explains as a
mitigating factor for a guideline violation.

Co-Authored-By: Warp Agent <agent@warp.dev>
@warp-agent-staging warp-agent-staging Bot added the factory-ab-eval Factory A/B evaluation replay label Aug 12, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

factory-ab-eval Factory A/B evaluation replay

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant