Add eval harness for testing AGENTS.md changes#69308

Open

RoyLee1224 wants to merge 16 commits into

apache:mainfrom

RoyLee1224:feat/skill-eval-harness

Commits on Jul 6, 2026

Add skill-eval harness scaffold with promptfoo
RoyLee1224
committed
ci: Add prek hook to remind eval on AGENTS.md and SKILL.md changes
RoyLee1224
committed
ci: add OAuth auth and runtime config generation to skill-eval harness
RoyLee1224
committed
feat: add skill-eval harness for AGENTS.md regression testing
RoyLee1224
committed
docs: update skill-eval README and use Helm routing as starter case
RoyLee1224
committed
fix: point skill-eval reminder hook at eval.py, not nonexistent eval.sh
RoyLee1224
committed
refactor: skip working arm when unchanged, show model in output
RoyLee1224
committed
refactor: skip working arm when unchanged, show model, use promptfoo@latest
RoyLee1224
committed
refactor: replace command-routing cases with newsfragment cases from real PRs
RoyLee1224
committed
fix: harden skill-eval harness and add prek hook tests
RoyLee1224
committed
feat: replace eval reminder hook with hash-based proof gate
RoyLee1224
committed
feat: run skill-eval via prek-managed node env, guard partial runs
RoyLee1224
committed
feat: follow the files/ output convention for eval results
RoyLee1224
committed
refactor: build promptfoo config as a dict, serialize to JSON
RoyLee1224
committed
refactor: Make prek the only entry point for the skill-eval
RoyLee1224
committed
Add positive newsfragment cases and view-skill-eval hook
RoyLee1224
committed