Skip to content

review-pr: harden Pre-Verdict Audit against comment-quality halo effect - #68

Draft
warp-agent-staging[bot] wants to merge 1 commit into
eval-base/019ff1e1from
factory/review-pr-audit-comment-halo-eval
Draft

review-pr: harden Pre-Verdict Audit against comment-quality halo effect#68
warp-agent-staging[bot] wants to merge 1 commit into
eval-base/019ff1e1from
factory/review-pr-audit-comment-halo-eval

Conversation

@warp-agent-staging

@warp-agent-staging warp-agent-staging Bot commented Aug 13, 2026

Copy link
Copy Markdown
Contributor

Summary

Hardens the ## Pre-Verdict Audit section of .agents/skills/review-pr/SKILL.md against a "comment-quality halo effect": a reviewer judged a comment as fine because it was well-written and explained a genuinely hard bug, and let that impression substitute for actually checking it against the target repo's commenting guidelines — missing confirmed rule violations as a result.

Changes

  • The Comments bullet in the Pre-Verdict Audit now requires enumerating every comment (doc comment or inline) added or changed by the diff individually, one by one with its file:line, instead of a single holistic pass over "every comment."
  • Each comment must be checked against the repository's own commenting guidelines when they exist, or — when the repository defines none — against the commenting distribution of the existing codebase (density, tone, what existing comments explain vs. omit).
  • Compliance must be evaluated independently of the comment's writing quality, technical accuracy, or how subtle/important the issue it describes is — none of those qualities excuses a violation of an applicable guideline or a clear mismatch with the codebase's own norms.
  • No other part of the skill was touched: schema, severity labels, safety rules, evidence rules, suggestion-block constraints, and the diff-line-annotation contract are all unchanged. The Tests bullet is unchanged.

Verification

  • git diff origin/eval-base/019ff1e1 touches exactly one file, .agents/skills/review-pr/SKILL.md, with a single line replaced (1 insertion, 1 deletion) — the Comments bullet under ## Pre-Verdict Audit. No other lines, trailing whitespace, or line endings changed.
  • Documentation-only change; no code paths affected.

Co-Authored-By: Warp Agent agent@warp.dev

Require enumerating each comment individually (file:line) during the
Pre-Verdict Audit so none get skimmed past as part of a holistic
impression, and explicitly forbid treating a comment's writing
quality, technical accuracy, or the importance of the issue it
describes as a mitigating factor when checking it against the
repository's commenting guidelines (or, absent guidelines, the
existing commenting distribution in the project).

Co-Authored-By: Warp Agent <agent@warp.dev>
@warp-agent-staging warp-agent-staging Bot added the factory-ab-eval Factory A/B evaluation replay label Aug 13, 2026
@warp-agent-staging

Copy link
Copy Markdown
Contributor Author

This PR was generated with Warp.

View run View conversation

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

factory-ab-eval Factory A/B evaluation replay

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant