Skip to content

review-pr: harden Pre-Verdict Audit against comment-quality halo effect - #58

Draft
warp-agent-staging[bot] wants to merge 1 commit into
eval-base/019ff1e1from
worker/pre-verdict-audit-halo-fix
Draft

review-pr: harden Pre-Verdict Audit against comment-quality halo effect#58
warp-agent-staging[bot] wants to merge 1 commit into
eval-base/019ff1e1from
worker/pre-verdict-audit-halo-fix

Conversation

@warp-agent-staging

Copy link
Copy Markdown
Contributor

Summary

Hardens the review-pr skill's ## Pre-Verdict Audit section against a "comment-quality halo effect": a reviewer previously let a comment's writing quality or technical accuracy ("well-written, explains a genuinely hard bug") substitute for actually checking it against the target repo's commenting guidelines, and missed confirmed rule violations as a result.

Change

Documentation-only edit to exactly one bullet under ## Pre-Verdict Audit in .agents/skills/review-pr/SKILL.md:

  • The Comments audit step now requires enumerating every added/changed comment individually, one by one with its file:line, instead of a holistic pass.
  • Each comment must be checked against the repo's own commenting guidelines, or — when the repo defines none — against the commenting distribution of existing code in the project (density, tone, what existing comments explain vs. omit).
  • Compliance must be evaluated independently of the comment's writing quality, technical accuracy, or how subtle/important the issue it describes is — none of those qualities excuses a violation.

No rule names or categories are introduced; the skill stays generic across repos with or without explicit written commenting guidelines.

Scope

Only the Comments bullet under ## Pre-Verdict Audit changed. The Tests bullet, the heading, its intro sentence, and everything else in the file (schema, severity labels, safety rules, evidence rules, suggestion-block constraints, diff-line-annotation contract) are untouched. Verified with git --no-pager diff: 1 file changed, 1 insertion(+), 1 deletion(-).

Conversation: https://staging.warp.dev/conversation/b4b70abe-a864-4588-a835-a1df9b950f9b
Run: https://oz.staging.warp.dev/runs/019ff4a4-6841-7c23-b9f0-14b13ca98138

This PR was generated with Oz.

Co-Authored-By: Warp Agent agent@warp.dev

Require enumerating each added/changed comment individually (file:line)
and checking it against the repo's own commenting guidelines, or against
the existing commenting distribution when no guidelines are defined.
Explicitly forbid treating a comment's writing quality, technical
accuracy, or the subtlety/importance of the issue it describes as a
mitigating factor for guideline violations.

Co-Authored-By: Warp Agent <agent@warp.dev>
@warp-agent-staging warp-agent-staging Bot added the factory-ab-eval Factory A/B evaluation replay label Aug 12, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

factory-ab-eval Factory A/B evaluation replay

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant