Skip to content

review-pr: harden Pre-Verdict Audit against comment-quality halo effect - #55

Draft
warp-agent-staging[bot] wants to merge 1 commit into
eval-base/019ff1e1from
factory/review-pr-audit-comment-halo
Draft

review-pr: harden Pre-Verdict Audit against comment-quality halo effect#55
warp-agent-staging[bot] wants to merge 1 commit into
eval-base/019ff1e1from
factory/review-pr-audit-comment-halo

Conversation

@warp-agent-staging

Copy link
Copy Markdown
Contributor

Overview

Hardens the ## Pre-Verdict Audit comment check in .agents/skills/review-pr/SKILL.md against a halo effect: a reviewer previously let a comment's writing quality and technical accuracy ("well-written, explains a genuinely hard bug") stand in for actually checking it against the target repo's commenting guidelines, and missed confirmed rule violations as a result.

Change

The **Comments** bullet now:

  • requires enumerating every added/changed comment (doc or inline) one by one with its file:line, so none get skimmed past as part of a holistic impression;
  • adds a fallback for repositories with no written commenting guidelines — judge against the commenting distribution of existing code (density, tone, what existing comments explain vs. omit);
  • explicitly forbids treating a comment's writing quality, technical accuracy, or the importance of the issue it describes as a mitigating factor for a guideline violation.

The guidance stays generic: no specific rule names or comment categories are introduced, so it still applies to repos with and without explicit commenting guidelines.

Scope

Documentation only, one line in one section. The schema, severity labels, safety rules, evidence rules, suggestion-block constraints, and diff-line-annotation contract are untouched, as is the **Tests** bullet and the section heading and intro.

Conversation: https://staging.warp.dev/conversation/1e1e1155-05d6-413b-b8bc-67ebdcd7910a
Run: https://oz.staging.warp.dev/runs/019ff3c6-db33-773a-bc56-b3f12b3ba741

This PR was generated with Oz.

Require the audit to enumerate each added/changed comment individually
with its file:line, fall back to the codebase's own commenting norms when
the repository defines no guidelines, and forbid treating a comment's
writing quality, technical accuracy, or importance as a mitigating
factor for a guideline violation.

Co-Authored-By: Warp <agent@warp.dev>
@warp-agent-staging warp-agent-staging Bot added the factory-ab-eval Factory A/B evaluation replay label Aug 12, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

factory-ab-eval Factory A/B evaluation replay

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant