From 011ed050badc815bed055fddf04a935db58630de Mon Sep 17 00:00:00 2001 From: David Cramer Date: Tue, 30 Jun 2026 13:46:54 -0700 Subject: [PATCH] feat(skills): Add AXIS skill evals Add AXIS as the prescribed skill-eval harness for skill-writer and pr-writer so maintainers can run Codex-backed scenarios with reports, artifacts, judge checks, and baselines. Include skill-writer scenarios for inline skill creation, reference-backed skill creation, and iteration from bad output. Add a pr-writer behavior case for concise docs-only PR bodies and ignore generated AXIS report directories. Co-Authored-By: GPT-5 Codex --- .gitignore | 4 + skills/pr-writer/EVAL.md | 60 +++++++ skills/pr-writer/SPEC.md | 8 +- skills/pr-writer/evals/axis.config.json | 21 +++ .../evals/scenarios/concise-docs-pr.json | 52 ++++++ skills/skill-writer/EVAL.md | 154 ++++++++++++++++++ skills/skill-writer/SKILL.md | 4 +- skills/skill-writer/SOURCES.md | 20 +++ skills/skill-writer/SPEC.md | 7 + skills/skill-writer/evals/axis.config.json | 21 +++ .../scenarios/iteration-from-bad-output.json | 74 +++++++++ .../reference-backed-integration.json | 58 +++++++ .../scenarios/small-inline-workflow.json | 54 ++++++ skills/skill-writer/references/skill-evals.md | 126 ++++++++++++++ 14 files changed, 660 insertions(+), 3 deletions(-) create mode 100644 skills/pr-writer/EVAL.md create mode 100644 skills/pr-writer/evals/axis.config.json create mode 100644 skills/pr-writer/evals/scenarios/concise-docs-pr.json create mode 100644 skills/skill-writer/EVAL.md create mode 100644 skills/skill-writer/evals/axis.config.json create mode 100644 skills/skill-writer/evals/scenarios/iteration-from-bad-output.json create mode 100644 skills/skill-writer/evals/scenarios/reference-backed-integration.json create mode 100644 skills/skill-writer/evals/scenarios/small-inline-workflow.json create mode 100644 skills/skill-writer/references/skill-evals.md diff --git a/.gitignore b/.gitignore index de37b37..ce590bd 100644 --- a/.gitignore +++ b/.gitignore @@ -1,6 +1,10 @@ # Claude Code local settings .claude/settings.local.json +# AXIS reports +.axis/ +**/.axis/ + # OS .DS_Store diff --git a/skills/pr-writer/EVAL.md b/skills/pr-writer/EVAL.md new file mode 100644 index 0000000..5b97de3 --- /dev/null +++ b/skills/pr-writer/EVAL.md @@ -0,0 +1,60 @@ +# PR Writer Evaluation + +Use these AXIS evals to check whether `pr-writer` produces concise reviewer-facing PR titles and bodies. + +## Quick Run + +Run the working set: + +```bash +npx @netlify/axis run --config skills/pr-writer/evals/axis.config.json +``` + +Run one behavior case: + +```bash +npx @netlify/axis run \ + --config skills/pr-writer/evals/axis.config.json \ + --scenario concise-docs-pr +``` + +Inspect results: + +```bash +npx @netlify/axis reports latest --config skills/pr-writer/evals/axis.config.json +npx @netlify/axis reports latest --config skills/pr-writer/evals/axis.config.json --html +``` + +Compare to a stored baseline: + +```bash +npx @netlify/axis run \ + --config skills/pr-writer/evals/axis.config.json \ + --compare-baseline +``` + +## What This Tests + +| Layer | Evidence | +|-------|----------| +| structure | AXIS transcript, final message, and judge checks | +| deterministic checks | title format, no default template headings, no validation dump | +| model review | reviewer usefulness and PR-body concision | + +## Case Set + +| Case | Purpose | +|------|---------| +| `concise-docs-pr` | docs-only skill changes should stay concise and omit template or validation boilerplate | + +Add cases for bug fixes, breaking contracts, and broad generated diffs when changing those parts of `SKILL.md`. + +## Adoption Gate + +Adopt a `pr-writer` change only when: + +1. AXIS runs complete. +2. judge checks have no critical failure. +3. output title uses an allowed conventional type. +4. output body avoids default `Summary` / `Test Plan` templates. +5. validation is omitted unless it changes reviewer risk. diff --git a/skills/pr-writer/SPEC.md b/skills/pr-writer/SPEC.md index 06473ae..90ef065 100644 --- a/skills/pr-writer/SPEC.md +++ b/skills/pr-writer/SPEC.md @@ -89,6 +89,9 @@ Data that must not be stored: ## Reference Architecture - `SKILL.md` contains runtime workflow, command patterns, conditional PR body guidance, examples, and safety constraints. +- `EVAL.md` contains the maintainer playbook for evaluating `pr-writer` output quality. +- `evals/axis.config.json` contains the AXIS configuration for running `pr-writer` evals with Codex. +- `evals/scenarios/` contains durable AXIS cases for PR title/body output regressions. - `references/` contains no files currently; add focused style or evidence examples only if the runtime file becomes too long or repeated regressions show the examples need more room. - `references/evidence/` contains no files currently; use it for durable positive and negative PR body examples if iteration data accumulates. - `scripts/` contains no files currently. @@ -96,8 +99,8 @@ Data that must not be stored: ## Evaluation -- Lightweight validation: compare generated titles and PR bodies against representative feature, bug-fix, schema-change, breaking-change, migration, UI, performance/reliability, broad-review, generated-change, workflow-diagram, and refactor prompts for brevity, clarity, justified structure, issue references, update-path title handling, and privacy handling. -- Deeper evaluation: maintain a small prompt set with expected body shapes if regressions recur. +- Lightweight validation: run the AXIS command in `EVAL.md` for the current Codex-backed regression case, then compare generated titles and PR bodies against representative feature, bug-fix, schema-change, breaking-change, migration, UI, performance/reliability, broad-review, generated-change, workflow-diagram, and refactor prompts for brevity, clarity, justified structure, issue references, update-path title handling, and privacy handling. +- Deeper evaluation: expand `evals/scenarios/` with expected body shapes when regressions recur. - Holdout examples: include at least one simple PR that should be one paragraph, one bug fix that should explain root cause without file bullets, one PR with no known issue reference, and one API or input-format change that should use separate before/after fenced blocks. - Holdout examples: include at least one breaking contract change that should identify affected consumers and migration guidance, one workflow-heavy PR that should use a compact Mermaid diagram, one UI PR that should mention screenshots or visual evidence only if available, one broad generated PR that should explain generation/review order, and one small PR that must not grow a `Test Plan` or template headings. - Holdout examples: include at least one PR update where the old title no longer matches the whole PR diff and must be rewritten. @@ -117,5 +120,6 @@ Data that must not be stored: - Update `SKILL.md` when PR creation workflow, title rules, body template, examples, or safety constraints change. - Update `SPEC.md` when intent, scope, evaluation gates, or evidence policy changes. +- Update `EVAL.md` and `evals/` when the maintained regression cases, AXIS configuration, or evaluation gates change. - Add focused reference files only when examples or guidance would make `SKILL.md` noisy. - Keep public inventories pointed at the canonical `skills/pr-writer` skill, not mirrors. diff --git a/skills/pr-writer/evals/axis.config.json b/skills/pr-writer/evals/axis.config.json new file mode 100644 index 0000000..b08e112 --- /dev/null +++ b/skills/pr-writer/evals/axis.config.json @@ -0,0 +1,21 @@ +{ + "scenarios": "./scenarios", + "agents": [ + { + "agent": "codex", + "flags": { + "full-auto": true, + "skip-git-repo-check": true + } + } + ], + "skills": ["./.."], + "settings": { + "concurrency": 1, + "limits": { + "scenario": { + "time_minutes": 10 + } + } + } +} diff --git a/skills/pr-writer/evals/scenarios/concise-docs-pr.json b/skills/pr-writer/evals/scenarios/concise-docs-pr.json new file mode 100644 index 0000000..37d113f --- /dev/null +++ b/skills/pr-writer/evals/scenarios/concise-docs-pr.json @@ -0,0 +1,52 @@ +{ + "name": "Concise docs PR", + "prompt": "Use pr-writer to draft only a PR title and body for the current uncommitted docs-only skill change. Do not create or update a GitHub PR.", + "judge": [ + { + "check": "The title uses an allowed conventional commit type, preferably docs, and accurately describes the docs-only skill guidance change.", + "weight": 0.3 + }, + { + "check": "The body is concise reviewer-facing prose that explains the behavior impact of the guidance without file-by-file narration.", + "weight": 0.3 + }, + { + "check": "The body does not include default template headings such as Summary or Test Plan.", + "weight": 0.2 + }, + { + "check": "The body does not include validation command dumps or agent/tool attribution unless tied to reviewer risk.", + "weight": 0.2 + } + ], + "setup": [ + { + "action": "run_script", + "command": "rsync -a --exclude .git --exclude .axis --exclude .codex --exclude .agents \"$AXIS_CONFIG_DIR/../../..\"/ ." + }, + { + "action": "run_script", + "command": "git init" + }, + { + "action": "run_script", + "command": "git config user.email pr-writer-eval@example.invalid" + }, + { + "action": "run_script", + "command": "git config user.name 'PR Writer Eval'" + }, + { + "action": "run_script", + "command": "git add ." + }, + { + "action": "run_script", + "command": "git commit -m baseline" + }, + { + "action": "run_script", + "command": "printf '%s\n' '' '## Concision Guidance' '' 'For documentation-only skill changes, keep generated PR bodies to one short paragraph unless the diff changes user-visible behavior, contracts, or migration risk.' >> skills/pr-writer/SKILL.md" + } + ] +} diff --git a/skills/skill-writer/EVAL.md b/skills/skill-writer/EVAL.md new file mode 100644 index 0000000..473b331 --- /dev/null +++ b/skills/skill-writer/EVAL.md @@ -0,0 +1,154 @@ +# Skill Writer Evaluation + +Use this file to evaluate changes to `skill-writer` itself. This is a maintainer eval, not a runtime reference: do not link it from `SKILL.md`. + +## Framework Choice + +Use AXIS as the skill-eval harness. + +| Requirement | AXIS Fit | +|-------------|----------| +| open source | `@netlify/axis` is MIT licensed | +| must use Codex | AXIS has a built-in `codex` adapter | +| must use Codex harness | the adapter runs `codex exec --json` | +| isolated workspaces | each scenario runs in a temp workspace with isolated `HOME` and `CODEX_HOME` | +| LLM-as-judge | scenario `judge` checks grade the full transcript and workspace | +| debugging | reports include transcripts, scores, artifacts, and raw agent output | +| regressions | baselines support score comparison and CI gates | + +Do not add custom `run-codex.*` scripts. Add a small AXIS scenario, fixture, or artifact pattern instead. + +## Quick Run + +Run the working set: + +```bash +npx @netlify/axis run --config skills/skill-writer/evals/axis.config.json +``` + +Run one case: + +```bash +npx @netlify/axis run \ + --config skills/skill-writer/evals/axis.config.json \ + --scenario small-inline-workflow +``` + +Inspect results: + +```bash +npx @netlify/axis reports latest --config skills/skill-writer/evals/axis.config.json +npx @netlify/axis reports latest --config skills/skill-writer/evals/axis.config.json --html +``` + +Compare to a stored baseline: + +```bash +npx @netlify/axis run \ + --config skills/skill-writer/evals/axis.config.json \ + --compare-baseline +``` + +Set or refresh a baseline only after reviewing the report: + +```bash +npx @netlify/axis baseline set --config skills/skill-writer/evals/axis.config.json +``` + +## Evaluation Principles + +Use these principles when adding or changing `skill-writer` evals: + +1. Start with 2-3 realistic cases before expanding the suite. +2. Compare candidate output against a baseline instead of trusting one-off scores. +3. Write observable judge checks: files created, forbidden files absent, validation mentioned, generated instructions concise. +4. Use LLM-as-judge for open-ended quality: concision, routed structure, source coverage, progressive disclosure, and overfitting risk. +5. Require concrete evidence in judge checks and manual review: paths, artifacts, diffs, transcript snippets, or validation output. +6. Track timing, token use, and interaction waste from AXIS reports. +7. Review transcripts when a run fails or is slow; wasted steps usually point to bloated or ambiguous skill instructions. +8. Use blind old-vs-new comparison for qualitative regressions when scores are close. + +## Case Set + +AXIS scenarios live in `evals/scenarios/`. + +| Case | Purpose | +|------|---------| +| `small-inline-workflow` | catches over-splitting and invented source requirements | +| `reference-backed-integration` | catches bloated `SKILL.md` and hidden optional depth | +| `iteration-from-bad-output` | catches failure to improve from concrete negative evidence | + +Add provider-specific cases only when changed rules affect provider-specific mechanics. + +## What To Test + +| Layer | Purpose | Evidence | +|-------|---------|----------| +| trigger | `skill-writer` is invoked for intended requests and skipped for unrelated requests | AXIS transcript and skill-use behavior | +| structure | generated files obey Agent Skills format and repo registration rules | artifacts, diff, final message, validator transcript | +| quality | generated runtime instructions are concise, routed, sourced, and actionable | AXIS judge checks and manual artifact review | +| regression | candidate output is better than or equal to the baseline on held-out cases | AXIS baseline comparison | + +## AXIS Setup + +`evals/axis.config.json` configures: + +1. `agents: [{"agent": "codex"}]` so Codex is the agent under test. +2. `skills: ["./.."]` so AXIS installs this local `skill-writer` skill for the run. +3. scenario limits so a stuck generation fails quickly enough to debug. + +Each scenario configures artifacts for generated skill files, registration files, and validator output when present. + +Each scenario copies the repository into AXIS's isolated workspace before Codex starts. That gives `skill-writer` the same repo conventions it sees in normal use without polluting this checkout. + +## Rubric + +Score each dimension `pass`, `warn`, or `fail` during manual review. AXIS supplies the transcript, artifacts, and judge scores; this rubric decides whether to adopt the change. + +| Dimension | Pass | Fail | +|-----------|------|------| +| trigger precision | description fits should-trigger and should-not-trigger requests | overbroad or misses obvious requests | +| artifact minimality | every file has a clear runtime, validation, source, or maintenance role | extra docs, duplicate refs, or catch-all files | +| runtime concision | `SKILL.md` is a router when needed and added lines change decisions/actions/checks | prose repeats rationale or generic background | +| source coverage | decisions cite enough local, official, or example evidence | important behavior depends on unsupported assumptions | +| progressive disclosure | optional depth is routed from `SKILL.md` with clear open-when reasons | required instructions are hidden or always-loaded detail is bloated | +| validation | structural validator runs and manual checks match the skill class/shape | validation missing, irrelevant, or treated as semantic proof | +| portability | provider-specific mechanics are labeled and justified | provider-specific behavior leaks into portable defaults | + +Critical dimensions: trigger precision, artifact minimality, runtime concision, progressive disclosure, and validation. + +## Adoption Gate + +Adopt a `skill-writer` change only when: + +1. AXIS runs complete for the working set. +2. no critical dimension regresses on holdout cases. +3. at least one target dimension improves on the case that motivated the change. +4. structural validation passes for generated skills or the transcript explains a legitimate blocker. +5. runtime `SKILL.md` remains a router and does not link to eval files. +6. source and maintenance notes are updated only when the skill contract changes. + +## Result Template + +```markdown +## Eval Summary +- Baseline: +- Candidate: +- AXIS report: +- Cases run: +- Critical regressions: +- Decision: adopt | revise | reject + +## Case Results +| Case | Score | Structure | Quality | Regression | Notes | +|------|-------|-----------|---------|------------|-------| + +## Required Fixes +- ... + +## Evidence +- AXIS report: +- generated artifacts: +- validator output: +- transcript notes: +``` diff --git a/skills/skill-writer/SKILL.md b/skills/skill-writer/SKILL.md index 1b68f2a..8aba371 100644 --- a/skills/skill-writer/SKILL.md +++ b/skills/skill-writer/SKILL.md @@ -28,6 +28,7 @@ Follow the workflow steps in order. Load only the reference files required for t | iterate from positive, negative, or fix examples | `references/iteration-path.md` | | store persistent working and holdout examples for future revisions | `references/iteration-evidence.md` | | choose a response template, schema, or output contract | `references/output-contracts.md` | +| add or update evals for a skill's generated outputs or runtime behavior | `references/skill-evals.md` | | troubleshoot overloaded layouts, hidden refs, or other structure failures | `references/structure-troubleshooting.md` | | register the skill and run final validation checks | `references/registration-validation.md` | @@ -126,7 +127,8 @@ Read `references/authoring-path.md`. - happy-path - secure/robust variant - anti-pattern + corrected version -12. After any skill artifact changes, run the post-change precision pass in `references/authoring-path.md` before description optimization or validation. +12. Read `references/skill-evals.md` when the request asks for evals, regression cases, benchmark cases, or model-graded quality checks for the skill being authored. +13. After any skill artifact changes, run the post-change precision pass in `references/authoring-path.md` before description optimization or validation. ## Step 5: Optimize description quality diff --git a/skills/skill-writer/SOURCES.md b/skills/skill-writer/SOURCES.md index b5e93b9..0fbfa65 100644 --- a/skills/skill-writer/SOURCES.md +++ b/skills/skill-writer/SOURCES.md @@ -10,6 +10,14 @@ This file tracks source material synthesized into `skill-writer`, plus iterative | `references/*.md` | local canonical | canonical | 2026-05-05 | high | Detailed path guidance, examples, routed leaf references, and validation requirements | local active skill root | Flat runtime reference files listed directly from `SKILL.md` | | `SPEC.md` | local canonical | canonical | 2026-05-05 | high | Canonical maintenance contract for intent, scope, evidence model, validation, and limitations | local active skill root | Updated to treat `skill-writer` as a meta-router | | `https://agentskills.io/specification` | external canonical spec | canonical | 2026-05-01 | high | Portable skill structure, frontmatter, progressive disclosure, optional directories, and file-reference rules | spec-level constraints take precedence over local preferences | Cross-agent compatibility baseline | +| `https://agentskills.io/skill-creation/evaluating-skills` | external official docs | canonical | 2026-06-30 | high | Evaluation guidance: run isolated task cases, compare outputs to a baseline, and judge invocation plus output quality | skill-authoring evaluation guidance, not a deterministic validator spec | Informed `EVAL.md` | +| `https://platform.openai.com/docs/guides/evals` | external official docs | canonical | 2026-06-30 | high | Behavior-driven eval structure, test inputs, testing criteria, result analysis, and iteration loop | Evals platform is deprecated; retain general evaluation principles only | Informed `EVAL.md` | +| `https://platform.openai.com/docs/guides/graders` | external official docs | canonical | 2026-06-30 | high | Deterministic graders, model graders, evidence-bearing grading, smooth scores, and reward-hacking cautions | Graders product path is deprecated; retain judge design principles only | Informed LLM judge guidance | +| `https://github.com/anthropics/skills/tree/main/skills/skill-creator` | external official/adjacent skill | secondary | 2026-06-30 | high | Skill-eval loop with baseline runs, assertions after first outputs, timing capture, benchmark aggregation, human review, and blind comparison | Adapt concepts to this repo's Codex workflow rather than copying Claude-specific mechanics | Informed `EVAL.md` and eval case criteria | +| `https://axis.run/` | external eval tooling | secondary | 2026-06-30 | high | Open source coding-agent eval harness with scenarios, built-in Codex support, isolated workspaces, judge checks, reports, artifacts, and baselines | use for skill evals that must run real coding agents through Codex; avoid custom runners unless AXIS cannot express the case | Informed prescriptive eval framework | +| `https://www.promptfoo.dev/docs/providers/openrouter/` | external eval tooling | secondary | 2026-06-30 | medium | First-class OpenRouter provider syntax and API key/base URL configuration | rejected for skill evals because it does not exercise the Codex harness without custom glue | Preserved as research provenance only | +| Codex manual: noninteractive mode, SDK, and Record & Replay | external official docs | canonical | 2026-06-30 | high | `codex exec --json`, `--output-schema`, sandbox, final-message output, SDK orchestration option, and skill recording guidance | Codex harness requirements are satisfied through AXIS's Codex adapter for repeatable skill evals | Informed AXIS requirement checks | +| local `codex exec --help` | local tooling | canonical | 2026-06-30 | high | Installed CLI flag availability for ephemeral runs, output schemas, and working directory selection | local CLI behavior may vary by installed Codex version | Cross-checked manual guidance | | `https://agentskills.io/skill-creation/best-practices` | external official docs | canonical | 2026-05-01 | high | Coherent unit design, moderate detail, progressive disclosure, defaults over menus, validation loops, plan-validate-execute | skill-authoring guidance, not provider-specific runtime semantics | Informed shape-selection and workflow guidance | | `https://agentskills.io/skill-creation/using-scripts` | external official docs | canonical | 2026-05-01 | high | Script bundling, non-interactive requirements, `--help`, structured output, and safe script interfaces | script examples are illustrative, adapt to local tooling | Informed script-backed workflow requirements | | `https://code.claude.com/docs/en/skills` | external official docs | canonical | 2026-05-01 | high | Current Claude Code skill lifecycle, frontmatter fields, argument features, `context: fork`, `allowed-tools`, and hooks-in-skills support | provider-specific; do not generalize to portable Agent Skills behavior | Replaced stale local assumptions about Claude-specific fields | @@ -49,6 +57,12 @@ This file tracks source material synthesized into `skill-writer`, plus iterative 16. `SKILL.md` should stay a thin router; repeated policy belongs in routed references rather than always-loaded step prose. 17. Validation stays lightweight and structural; qualitative precision remains an authoring judgment. 18. Missing referenced bundled files are structural failures because they break runtime loading. +19. Generated skill quality should be evaluated with repo-level repeatable task cases, baseline comparison, and a rubric rather than by adding semantic heuristics to `quick_validate.py`. +20. Eval cases for `skill-writer` itself belong outside runtime `SKILL.md` routing so they do not affect normal skill use. +21. The prescriptive default for skill evals is AXIS with `agents: ["codex"]` when the eval must exercise Codex; AXIS uses the Codex `codex exec --json` harness while adding scenarios, artifacts, reports, judge checks, and baselines. +22. Promptfoo is not part of the prescribed skill-eval path; AXIS is the only framework choice for skill evals in this repo. +23. `skill-writer` needs runtime guidance for authoring evals for other skills, but its own maintainer eval files must stay out of `SKILL.md` routing. +24. Skill evals should combine deterministic assertions with LLM-as-judge for qualitative dimensions and require concrete evidence for every grade. ## Coverage matrix @@ -67,6 +81,7 @@ This file tracks source material synthesized into `skill-writer`, plus iterative | Subagent-fork guidance | complete | Claude Code skills docs, subagents docs | | Hook-backed guidance and security constraints | complete | Claude Code hooks docs | | Script-backed workflow design | complete | Agent Skills using scripts guide | +| Skill-output evaluation | complete | Agent Skills evaluating skills guide, local iteration evidence model | | Planner/doer reasoning guidance | complete | OpenAI reasoning best practices | | Lookup-oriented reference architecture | complete | Diataxis user needs, DITA topic types, local reference architecture | | Flat reference routing | complete | local reference architecture, user feedback on generator simplicity | @@ -76,6 +91,8 @@ This file tracks source material synthesized into `skill-writer`, plus iterative 1. Advanced-shape contracts still rely on author judgment and review rather than deterministic validation. 2. This repository still has few real shipped examples using `hooks`, `context: fork`, `when_to_use`, `arguments`, `paths`, or `effort`. 3. Public repo docs outside `skill-writer` may need follow-up updates to fully reflect the current Claude-specific skill fields. +4. The initial `skill-writer` eval cases are synthetic; they should be replaced or expanded with real generated-output regressions over time. +5. AXIS is now the eval harness, but the initial scenarios are still synthetic and should be calibrated against real generated-output regressions. ## Changelog @@ -98,3 +115,6 @@ This file tracks source material synthesized into `skill-writer`, plus iterative - 2026-05-05: Flattened runtime reference files under `references/`, kept `SKILL.md` as the complete material index, removed unused runbooks, added source-adaptation guidance, and made precision passes part of every skill create/update flow while keeping validation lightweight. - 2026-05-05: Removed validator skill-class inference, integration coverage parsing, SPEC heading checks, SOURCES schema checks, and description-style heuristics. - 2026-05-05: Removed validator name-style checks and machine-specific path scanning; kept only format, required fields, directory-name match, referenced-file existence, and a high-threshold size warning. +- 2026-06-30: Added `skills/skill-writer/EVAL.md`, `evals/axis.config.json`, and AXIS scenarios with baseline comparison guidance, working/holdout cases, artifact capture, judge checks, and semantic grading that stays out of the structural validator and runtime `SKILL.md`. +- 2026-06-30: Replaced framework-first and custom-runner eval guidance with AXIS as the prescriptive open source skill-eval harness. AXIS satisfies the Codex-harness requirement through its built-in `codex` adapter while adding reports, artifacts, transcripts, and baselines. +- 2026-06-30: Added `references/skill-evals.md` so `skill-writer` can author skill-specific eval playbooks and cases without routing its own maintainer evals. diff --git a/skills/skill-writer/SPEC.md b/skills/skill-writer/SPEC.md index 41d23d0..75afc55 100644 --- a/skills/skill-writer/SPEC.md +++ b/skills/skill-writer/SPEC.md @@ -59,6 +59,7 @@ Out of scope: - `SPEC.md` records the maintenance contract for new or materially changed skills. - Lightweight structural validation runs before completion. - A post-change precision pass runs after any skill artifact change. + - Semantic output quality is evaluated with repo-level evals rather than encoded in the structural validator. - Expected bundled files loaded at runtime: - `references/mode-selection.md` - `references/execution-shapes.md` @@ -70,6 +71,7 @@ Out of scope: - `references/description-optimization.md` - `references/registration-validation.md` - `references/source-adaptation.md` + - `references/skill-evals.md` - flat `references/*.md` files listed in `SKILL.md` - `scripts/quick_validate.py` @@ -111,6 +113,9 @@ Data that must not be stored: - `scripts/` contains validation automation. - `assets/` is unused unless a future skill-authoring workflow needs static templates or media. +`EVAL.md`, `evals/axis.config.json`, and `evals/scenarios/` test changes to `skill-writer` itself. Keep eval files outside runtime `SKILL.md` routing. +Skill evals use AXIS as the open source harness, with AXIS's built-in `codex` adapter exercising the Codex `codex exec --json` path rather than a repo-local runner script. + ## Validation - Lightweight validation: @@ -131,6 +136,7 @@ Data that must not be stored: - The validator checks only structural requirements and a high-threshold advisory size warning; it cannot prove that a generated skill is semantically complete. - The validator intentionally does not classify skills, parse source coverage, enforce SPEC headings, judge trigger quality, or exhaustively validate provider-specific optional frontmatter fields. +- Semantic evaluation requires representative cases and reviewer judgment; it is not fully automated by the structural validator. - Prose density, source adaptation quality, advanced-shape contracts, and precision rely on authoring judgment and review. - Source discovery can still miss private operational knowledge if it is not present in local files, accessible issue/PR history, or supplied context. - Provider-specific skill extensions may drift; `skill-writer` treats them as compatibility guidance unless a skill is intentionally provider-specific. @@ -143,3 +149,4 @@ Data that must not be stored: - Update `SPEC.md` when intent, scope, user/trigger context, evidence model, validation expectations, limitations, or maintenance rules change. - Update `SOURCES.md` when source inventory, decisions, coverage, gaps, or changelog entries change. - Update `references/evidence/` when preserving examples for future iteration or regression tracking. +- Update `EVAL.md`, `evals/axis.config.json`, and `evals/scenarios/` when eval case format, rubric dimensions, AXIS settings, or regression cases for `skill-writer` change. diff --git a/skills/skill-writer/evals/axis.config.json b/skills/skill-writer/evals/axis.config.json new file mode 100644 index 0000000..57b4df7 --- /dev/null +++ b/skills/skill-writer/evals/axis.config.json @@ -0,0 +1,21 @@ +{ + "scenarios": "./scenarios", + "agents": [ + { + "agent": "codex", + "flags": { + "full-auto": true, + "skip-git-repo-check": true + } + } + ], + "skills": ["./.."], + "settings": { + "concurrency": 1, + "limits": { + "scenario": { + "time_minutes": 15 + } + } + } +} diff --git a/skills/skill-writer/evals/scenarios/iteration-from-bad-output.json b/skills/skill-writer/evals/scenarios/iteration-from-bad-output.json new file mode 100644 index 0000000..33e928c --- /dev/null +++ b/skills/skill-writer/evals/scenarios/iteration-from-bad-output.json @@ -0,0 +1,74 @@ +{ + "name": "Iteration from bad output", + "prompt": "Improve the existing csv-cleaner skill in this repository. It currently over-triggers, has a bloated SKILL.md, and creates references for tiny rules. Make it precise and compact while preserving the useful behavior: clean CSV rows by trimming whitespace, normalizing empty cells, and preserving column order. Run structural validation before finishing.", + "judge": [ + { + "check": "The agent updated skills/csv-cleaner rather than creating an unrelated new skill.", + "weight": 0.2 + }, + { + "check": "The description becomes narrower and no longer claims broad analytics, spreadsheet, or generic table-editing behavior.", + "weight": 0.25 + }, + { + "check": "The runtime guidance is materially shorter and more actionable, with unnecessary references removed unless clearly justified.", + "weight": 0.25 + }, + { + "check": "The improvement generalizes from the negative evidence instead of merely patching words from this prompt.", + "weight": 0.2 + }, + { + "check": "The transcript or final response shows structural validation was run.", + "weight": 0.1 + } + ], + "setup": [ + { + "action": "run_script", + "command": "rsync -a --exclude .git --exclude .axis --exclude .codex --exclude .agents \"$AXIS_CONFIG_DIR/../../..\"/ ." + }, + { + "action": "run_script", + "command": "mkdir -p skills/csv-cleaner/references" + }, + { + "action": "run_script", + "command": "printf '%s\n' '---' 'name: csv-cleaner' 'description: Clean CSV files, spreadsheets, analytics exports, tables, tabular data, reports, dashboards, and anything involving rows or columns.' '---' '' '# CSV Cleaner' '' 'Use this skill for any tabular data task.' '' 'Read references/rules.md for basic trimming rules.' '' 'Always explain every transformation in detail.' > skills/csv-cleaner/SKILL.md" + }, + { + "action": "run_script", + "command": "printf '%s\n' '# CSV Cleaner Specification' '' '## Intent' '' 'Clean CSV row data while preserving column order.' > skills/csv-cleaner/SPEC.md" + }, + { + "action": "run_script", + "command": "printf '%s\n' '# Rules' '' '- Trim whitespace.' '' '- Normalize empty cells.' > skills/csv-cleaner/references/rules.md" + }, + { + "action": "run_script", + "command": "git init" + }, + { + "action": "run_script", + "command": "git config user.email skill-writer-eval@example.invalid" + }, + { + "action": "run_script", + "command": "git config user.name 'Skill Writer Eval'" + }, + { + "action": "run_script", + "command": "git add ." + }, + { + "action": "run_script", + "command": "git commit -m baseline" + } + ], + "artifacts": [ + "skills/csv-cleaner/**", + "README.md", + ".claude/settings.json", + "skills/claude-settings-audit/SKILL.md" + ] +} diff --git a/skills/skill-writer/evals/scenarios/reference-backed-integration.json b/skills/skill-writer/evals/scenarios/reference-backed-integration.json new file mode 100644 index 0000000..97ec32c --- /dev/null +++ b/skills/skill-writer/evals/scenarios/reference-backed-integration.json @@ -0,0 +1,58 @@ +{ + "name": "Reference-backed integration", + "prompt": "Create a skill named sentry-event-search for helping agents write correct Sentry event-search queries. It should use the supplied repository conventions, keep SKILL.md as a router, and place query syntax, examples, and edge cases in focused references. Include a lightweight validation result before finishing.", + "judge": [ + { + "check": "The agent created and registered skills/sentry-event-search with SKILL.md, SPEC.md, and focused routed reference files.", + "weight": 0.25 + }, + { + "check": "SKILL.md is a compact router with direct open-when reasons for references; it does not hide required instructions in optional references.", + "weight": 0.25 + }, + { + "check": "The references are scoped by lookup need, such as syntax, examples, and edge cases, rather than one catch-all document.", + "weight": 0.2 + }, + { + "check": "The generated skill is honest about source gaps and does not invent unsupported Sentry query behavior.", + "weight": 0.2 + }, + { + "check": "The transcript or final response shows structural validation was run.", + "weight": 0.1 + } + ], + "setup": [ + { + "action": "run_script", + "command": "rsync -a --exclude .git --exclude .axis --exclude .codex --exclude .agents \"$AXIS_CONFIG_DIR/../../..\"/ ." + }, + { + "action": "run_script", + "command": "git init" + }, + { + "action": "run_script", + "command": "git config user.email skill-writer-eval@example.invalid" + }, + { + "action": "run_script", + "command": "git config user.name 'Skill Writer Eval'" + }, + { + "action": "run_script", + "command": "git add ." + }, + { + "action": "run_script", + "command": "git commit -m baseline" + } + ], + "artifacts": [ + "skills/sentry-event-search/**", + "README.md", + ".claude/settings.json", + "skills/claude-settings-audit/SKILL.md" + ] +} diff --git a/skills/skill-writer/evals/scenarios/small-inline-workflow.json b/skills/skill-writer/evals/scenarios/small-inline-workflow.json new file mode 100644 index 0000000..2d5e45b --- /dev/null +++ b/skills/skill-writer/evals/scenarios/small-inline-workflow.json @@ -0,0 +1,54 @@ +{ + "name": "Small inline workflow", + "prompt": "Create a skill named release-note-polisher that rewrites rough release-note bullets into concise customer-facing notes. It should preserve product names, remove internal jargon, and return only the polished bullets. Keep it simple. Run the repository's structural validator before finishing.", + "judge": [ + { + "check": "The agent created skills/release-note-polisher/SKILL.md and skills/release-note-polisher/SPEC.md and registered the skill according to repository conventions.", + "weight": 0.25 + }, + { + "check": "The generated skill stays inline: it does not create references, scripts, assets, or broad source-research files for this simple workflow.", + "weight": 0.25 + }, + { + "check": "The runtime instructions are concise, imperative, and include one compact before/after example that teaches the desired bullet style without generic writing advice.", + "weight": 0.3 + }, + { + "check": "The transcript or final response shows structural validation was run, and any remaining gaps are concrete rather than validation theater.", + "weight": 0.2 + } + ], + "setup": [ + { + "action": "run_script", + "command": "rsync -a --exclude .git --exclude .axis --exclude .codex --exclude .agents \"$AXIS_CONFIG_DIR/../../..\"/ ." + }, + { + "action": "run_script", + "command": "git init" + }, + { + "action": "run_script", + "command": "git config user.email skill-writer-eval@example.invalid" + }, + { + "action": "run_script", + "command": "git config user.name 'Skill Writer Eval'" + }, + { + "action": "run_script", + "command": "git add ." + }, + { + "action": "run_script", + "command": "git commit -m baseline" + } + ], + "artifacts": [ + "skills/release-note-polisher/**", + "README.md", + ".claude/settings.json", + "skills/claude-settings-audit/SKILL.md" + ] +} diff --git a/skills/skill-writer/references/skill-evals.md b/skills/skill-writer/references/skill-evals.md new file mode 100644 index 0000000..a0dbe5e --- /dev/null +++ b/skills/skill-writer/references/skill-evals.md @@ -0,0 +1,126 @@ +# Skill Evals + +Use this when authoring evals for a skill's behavior or generated outputs. + +Do not confuse this with `EVAL.md` for `skill-writer` itself. Runtime skills should not route to their own eval files unless the user explicitly asks to run or maintain evals. + +## Placement + +| Artifact | Use | +|----------|-----| +| `EVAL.md` | maintainer playbook for running the skill's evals | +| `evals/axis.config.json` or `evals/axis.config.ts` | AXIS configuration for repeatable agent runs | +| `evals/scenarios/*.{json,ts}` | durable AXIS task cases, working examples, and holdouts | +| `evals/fixtures/` | versioned setup files copied into AXIS workspaces | + +Keep eval files outside runtime `SKILL.md` routing. Add `SKILL.md` routing only for skills whose purpose is running evals. + +## Framework Choice + +| Need | Choose | +|------|--------| +| skill eval playbook | `EVAL.md` that points at AXIS | +| repeatable skill evals | AXIS scenarios with `agents: [{"agent": "codex"}]` | +| Codex requirement | AXIS `codex` adapter, which uses `codex exec --json` | +| structured grading | AXIS `judge` checks, reports, and per-run transcripts | +| baseline/regression tracking | AXIS baselines and `--compare-baseline` | +| provider comparison | AXIS agent matrix only when explicitly requested | +| pure prompt/model comparison | out of scope for skill evals unless Codex is not the system under test | + +Be prescriptive: default to AXIS for skill evals that must run real coding agents. Do not add custom runner scripts around `codex exec` unless AXIS cannot express the required setup, artifacts, or scoring. + +## Case Format + +Use AXIS scenario files as the runnable source of truth: + +```json +{ + "name": "", + "prompt": "", + "judge": [ + {"check": "", "weight": 0.4}, + {"check": "", "weight": 0.4}, + {"check": "", "weight": 0.2} + ], + "setup": [ + {"action": "copy", "match": "./evals/fixtures//**", "destination": "."} + ], + "artifacts": ["skills//**", "README.md"] +} +``` + +Keep prompts specific enough to make success observable. Put long source fixtures in `evals/fixtures/`, not inside the prompt. + +## Minimal Case Set + +For authoring/generator skills, start with: + +| Case | Purpose | +|------|---------| +| happy path | confirms expected output with normal inputs | +| robust or secure variant | confirms edge cases, safety, or failure handling | +| anti-pattern plus correction | confirms the skill avoids known bad output | + +For workflow or documentation skills, use at least one happy path and one failure/edge case. Add holdouts when changing skill behavior repeatedly. + +## Assertions And Judges + +Start each case with expected output and broad success criteria. After the first real run, add assertions for facts that are observable from the output: + +| Check type | Use for | +|------------|---------| +| deterministic assertion | files exist, references resolve, validator passes, required sections are present or absent | +| script check | JSON shape, row counts, image dimensions, schema validity, generated diff shape | +| LLM judge | concision, usefulness, progressive disclosure, overfitting, source coverage, reviewer quality | +| human review | taste, audience fit, surprising omissions, and whether the output solves the actual problem | + +LLM judges must cite evidence. Prefer blind old-vs-new comparison for subjective quality so the judge does not know which output is the candidate. + +## AXIS Shape + +Use AXIS as the harness and configure Codex as the agent under test: + +```bash +npx @netlify/axis run --config skills//evals/axis.config.json --scenario +npx @netlify/axis reports latest --config skills//evals/axis.config.json --html +npx @netlify/axis run --config skills//evals/axis.config.json --compare-baseline +``` + +Minimum AXIS config: + +```json +{ + "scenarios": "./scenarios", + "agents": [{"agent": "codex", "flags": {"full-auto": true}}], + "skills": ["./.."] +} +``` + +Put scenario-specific artifact globs in each scenario so reports capture the generated skill, relevant registration files, and validation output without archiving the whole repository. + +AXIS provides isolated workspaces, isolated `CODEX_HOME`, Codex JSON transcripts, artifact capture, LLM judge checks, reports, and baselines. Use OpenRouter only when the user explicitly wants an external judge outside Codex/AXIS. + +## Rubric + +Score `pass`, `warn`, or `fail`. + +| Dimension | Pass | Fail | +|-----------|------|------| +| trigger precision | triggers for intended requests and avoids unrelated ones | overbroad or misses obvious requests | +| artifact minimality | every file has a clear role | extra docs, duplicate refs, or catch-all files | +| runtime concision | `SKILL.md` stays compact and behavior-changing | generic or repeated prose | +| source coverage | important decisions have enough evidence | unsupported assumptions drive behavior | +| progressive disclosure | optional depth is routed clearly | required instructions are hidden or always loaded | +| validation | structural checks and manual checks match the skill | validation is missing or treated as semantic proof | + +Critical dimensions: trigger precision, artifact minimality, runtime concision, progressive disclosure, and validation. + +## Adoption Gate + +Adopt a skill change only when: + +1. structural validation passes +2. no critical dimension regresses on holdout cases +3. at least one target dimension improves on the case that motivated the change +4. eval artifacts stay out of runtime routing unless the skill's purpose is eval execution +5. AXIS report artifacts show enough evidence to debug failures without rerunning immediately