Skip to content

review-pr: harden Pre-Verdict Audit against comment-quality halo effect - #54

Draft
warp-agent-staging[bot] wants to merge 1 commit into
eval-base/019ff1e1from
factory/review-pr-audit-halo
Draft

review-pr: harden Pre-Verdict Audit against comment-quality halo effect#54
warp-agent-staging[bot] wants to merge 1 commit into
eval-base/019ff1e1from
factory/review-pr-audit-halo

Conversation

@warp-agent-staging

Copy link
Copy Markdown
Contributor

What

Rewrites the - **Comments**: bullet under ## Pre-Verdict Audit in .agents/skills/review-pr/SKILL.md. Nothing else in the file changes.

The audit now requires the reviewer to:

  • Enumerate every added/changed comment (doc or inline) one by one with its file:line, so none get skimmed past as part of a holistic impression.
  • Check each one individually against the repository's commenting guidelines — or, when the repository defines none, against the commenting distribution of the existing code (density, tone, what existing comments explain vs. omit).
  • Judge compliance independently of the comment's writing quality, technical accuracy, or the importance of the issue it describes; none of those excuse a guideline violation.

Why

A reviewer using this skill let a comment's writing quality and technical accuracy ("well-written, explains a genuinely hard bug") stand in for actually checking it against the target repo's commenting guidelines, and missed confirmed rule violations as a result. Per-comment enumeration removes the halo effect, and the explicit prohibition removes the excuse.

The wording stays generic on purpose: no specific rule names or categories are introduced, so the skill still applies to repositories with or without written commenting guidelines.

Scope

One line changed. Schema, severity labels, safety rules, evidence rules, suggestion-block constraints, and the diff-line-annotation contract are untouched.

Conversation: https://staging.warp.dev/conversation/5e46bb2e-9a34-43ae-862f-bb18d4c1572d
Run: https://oz.staging.warp.dev/runs/019ff3c6-d39c-7218-a1c8-33839d68b1a9

This PR was generated with Oz.

Require the audit to enumerate each added/changed comment individually
with its file:line, and forbid treating a comment's writing quality,
technical accuracy, or the importance of the issue it describes as a
mitigating factor for a guideline violation. Also covers repositories
with no written commenting guidelines by falling back to the existing
codebase's own commenting norms.

Co-Authored-By: Warp <agent@warp.dev>
@warp-agent-staging warp-agent-staging Bot added the factory-ab-eval Factory A/B evaluation replay label Aug 12, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

factory-ab-eval Factory A/B evaluation replay

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant