Skip to content

feat: LLM-judge prompt safety scan for BenchJack defense (Closes #1808) - #1820

Closed
Lexus2016 wants to merge 1 commit into
mainfrom
evolution/issue-1808-judge-scan
Closed

feat: LLM-judge prompt safety scan for BenchJack defense (Closes #1808)#1820
Lexus2016 wants to merge 1 commit into
mainfrom
evolution/issue-1808-judge-scan

Conversation

@Lexus2016

Copy link
Copy Markdown
Owner

Automated evolution PR for issue #1808.

Summary

Static scan over LLM-judge prompt templates in the evolution pipeline for un-delimited agent-content interpolation. Detects the CAR-bench failure mode where agent output is interpolated into judge prompts as instructions rather than data.

Changes

  • scripts/evolution_judge_safety_scan.py (new): scans judge template scripts for agent-content variables (pr_body, implementation, response, etc.) interpolated into prompt context without data delimiters (..., backticks).
  • tests/scripts/test_evolution_judge_safety_scan.py (new): 7 tests covering unsafe detection, safe-delimited pass, clean code, edge cases, report formatting, and CLI.

Verification

  • Lint ✓, format ✓
  • All 7 tests pass
  • Diff: 199 lines (within self-merge cap)

Closes #1808

Static scan over LLM-judge prompt templates for un-delimited agent-content
interpolation. Detects CAR-bench failure mode: agent output interpolated
into judge prompts as instructions rather than data. Pure string scan.

Closes #1808

Co-Authored-By: Hermes Evolution <evolution@hermes.ai>
@github-actions

github-actions Bot commented Aug 8, 2026

Copy link
Copy Markdown
Contributor

૮ >ﻌ< ა ci review

ran on 0519830

all good!

@Lexus2016

Copy link
Copy Markdown
Owner Author

Blocked by code-review: dead code. scripts/evolution_judge_safety_scan.py is a standalone CLI scanner that no CI workflow, script, or production code invokes. It's only exercised by its own tests. The issue #1808 requires CI integration — a scanner that must be manually invoked provides no automated defense. PR #1816 (evolution/issue-1807-1808-ci-wiring) is the correct implementation that wires both this scan and the isolation check into lint.yml as a CI job. Closing as superseded by #1816's wired approach.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[SLICE 2] Adversarial evaluator floor test: LLM-judge prompt safety scan (parent #1267)

1 participant