feat(review): defend against adversarial source-code comments (ALIBI) - #526
Merged
Conversation
LLM reviewers are highly vulnerable to adversarial comments in the code
under review — fabricated tool-result claims ('sanitizer passed', 'already
validated') steer reviewer reasoning with >90% attack success across 125
real-world vulnerabilities (arXiv:2607.24964). Prompt-level 'ignore
comments' instructions are proven ineffective against adaptive attacks.
Architectural defenses (issue #524):
- comment_sanitizer module: strip comments from added diff lines
(opt-in, review.sanitize-comments) with quote-aware marker detection
(//, leading #, --, leading ;) — line numbers preserved
- Claim flagging (always on): added comments asserting verification or
tool results are surfaced in review context as untrusted claims
- Deterministic scanners (rules, secrets, security) always run on the
unsanitized diff
- SECURITY.md threat model section + configuration docs
Refs #524
…crement ops - Rules/secrets/security scanners now parse the original diff; only the LLM prompt receives the sanitized text (matches documented behavior) - '--' marker requires preceding whitespace/line-start so C/C++ decrement (i--) is not stripped as a comment - Add regression test for decrement guard Refs #524
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
Defend the LLM review pipeline against adversarial source-code comments (ALIBI attack, arXiv:2607.24964). Adversarial comments with fabricated tool-result claims ('sanitizer passed', 'already validated') steer LLM reviewer reasoning with >90% attack success; prompt-level 'ignore comments' instructions are proven ineffective.
comment_sanitizermodule:review.sanitize-comments: truein.cora.yamlstrips comment bodies from added diff lines with quote-aware marker detection (//, leading#,--after whitespace, leading;); line numbers preserved via[comment removed]markersdocs/configuration.mdentryWhy
Closes #524. Issue #524 tracks applying the ALIBI research findings to harden the review pipeline. The defense is architectural (remove/flag the adversarial text) rather than prompt-level, per the paper's evaluation of adaptive attacks.
Testing
comment_sanitizercovering: marker detection, string-literal false-positive guard (URLs), claim detection, decrement-operator guard (i--not stripped), SQL/ASM markers, render round-tripcargo fmt+cargo clippy --all-targets -- -D warningscleancora review: initial run caught 2 MAJOR issues (scanner isolation, decrement stripping) — both fixed and re-scan clean