Security: harden review pipeline against adversarial code comments (ALIBI, arXiv:2607.24964)

[ajianaz]

Context

ALIBI shows LLM-based vulnerability detectors & code reviewers are highly vulnerable to adversarial source-code comments that steer detector reasoning without changing program behavior (attack success >90% across 125 real-world null-pointer vulns, 100% on one system). Most effective: comments steering reasoning or fabricating external-tool results (e.g. claiming a sanitizer check already passed).

Key finding: prompt-level defenses provide limited robustness against adaptive attacks; architectural isolation and pre-detector comment sanitization substantially improve resilience.

Proposal

• Optional sanitize-comments mode in the review pipeline: strip comments from diff context before sending to the LLM
• Heuristic flagging: elevate severity when a diff adds comments claiming verification/tool results (e.g. 'already validated', 'sanitizer passed', 'tested by X')
• Do NOT rely on system-prompt instructions to ignore such comments — proven ineffective
• Document the threat model in security docs

References

• Paper: http://arxiv.org/abs/2607.24964