feat(skills): Add AXIS skill evals

.gitignore

@@ -1,6 +1,10 @@
 # Claude Code local settings
 .claude/settings.local.json
 
+# AXIS reports
+.axis/
+**/.axis/
+
 # OS
 .DS_Store
 

skills/pr-writer/EVAL.md

@@ -0,0 +1,60 @@
+# PR Writer Evaluation
+
+Use these AXIS evals to check whether `pr-writer` produces concise reviewer-facing PR titles and bodies.
+
+## Quick Run
+
+Run the working set:
+
+```bash
+npx @netlify/axis run --config skills/pr-writer/evals/axis.config.json
+```
+
+Run one behavior case:
+
+```bash
+npx @netlify/axis run \
+  --config skills/pr-writer/evals/axis.config.json \
+  --scenario concise-docs-pr
+```
+
+Inspect results:
+
+```bash
+npx @netlify/axis reports latest --config skills/pr-writer/evals/axis.config.json
+npx @netlify/axis reports latest --config skills/pr-writer/evals/axis.config.json --html
+```
+
+Compare to a stored baseline:
+
+```bash
+npx @netlify/axis run \
+  --config skills/pr-writer/evals/axis.config.json \
+  --compare-baseline
+```
+
+## What This Tests
+
+| Layer | Evidence |
+|-------|----------|
+| structure | AXIS transcript, final message, and judge checks |
+| deterministic checks | title format, no default template headings, no validation dump |
+| model review | reviewer usefulness and PR-body concision |
+
+## Case Set
+
+| Case | Purpose |
+|------|---------|
+| `concise-docs-pr` | docs-only skill changes should stay concise and omit template or validation boilerplate |
+
+Add cases for bug fixes, breaking contracts, and broad generated diffs when changing those parts of `SKILL.md`.
+
+## Adoption Gate
+
+Adopt a `pr-writer` change only when:
+
+1. AXIS runs complete.
+2. judge checks have no critical failure.
+3. output title uses an allowed conventional type.
+4. output body avoids default `Summary` / `Test Plan` templates.
+5. validation is omitted unless it changes reviewer risk.

skills/pr-writer/SPEC.md

@@ -89,15 +89,18 @@ Data that must not be stored:
 ## Reference Architecture
 
 - `SKILL.md` contains runtime workflow, command patterns, conditional PR body guidance, examples, and safety constraints.
+- `EVAL.md` contains the maintainer playbook for evaluating `pr-writer` output quality.
+- `evals/axis.config.json` contains the AXIS configuration for running `pr-writer` evals with Codex.
+- `evals/scenarios/` contains durable AXIS cases for PR title/body output regressions.
 - `references/` contains no files currently; add focused style or evidence examples only if the runtime file becomes too long or repeated regressions show the examples need more room.
 - `references/evidence/` contains no files currently; use it for durable positive and negative PR body examples if iteration data accumulates.
 - `scripts/` contains no files currently.
 - `assets/` contains no files currently.
 
 ## Evaluation
 
-- Lightweight validation: compare generated titles and PR bodies against representative feature, bug-fix, schema-change, breaking-change, migration, UI, performance/reliability, broad-review, generated-change, workflow-diagram, and refactor prompts for brevity, clarity, justified structure, issue references, update-path title handling, and privacy handling.
-- Deeper evaluation: maintain a small prompt set with expected body shapes if regressions recur.
+- Lightweight validation: run the AXIS command in `EVAL.md` for the current Codex-backed regression case, then compare generated titles and PR bodies against representative feature, bug-fix, schema-change, breaking-change, migration, UI, performance/reliability, broad-review, generated-change, workflow-diagram, and refactor prompts for brevity, clarity, justified structure, issue references, update-path title handling, and privacy handling.
+- Deeper evaluation: expand `evals/scenarios/` with expected body shapes when regressions recur.
 - Holdout examples: include at least one simple PR that should be one paragraph, one bug fix that should explain root cause without file bullets, one PR with no known issue reference, and one API or input-format change that should use separate before/after fenced blocks.
 - Holdout examples: include at least one breaking contract change that should identify affected consumers and migration guidance, one workflow-heavy PR that should use a compact Mermaid diagram, one UI PR that should mention screenshots or visual evidence only if available, one broad generated PR that should explain generation/review order, and one small PR that must not grow a `Test Plan` or template headings.
 - Holdout examples: include at least one PR update where the old title no longer matches the whole PR diff and must be rewritten.
@@ -117,5 +120,6 @@ Data that must not be stored:
 
 - Update `SKILL.md` when PR creation workflow, title rules, body template, examples, or safety constraints change.
 - Update `SPEC.md` when intent, scope, evaluation gates, or evidence policy changes.
+- Update `EVAL.md` and `evals/` when the maintained regression cases, AXIS configuration, or evaluation gates change.
 - Add focused reference files only when examples or guidance would make `SKILL.md` noisy.
 - Keep public inventories pointed at the canonical `skills/pr-writer` skill, not mirrors.

skills/pr-writer/evals/axis.config.json

@@ -0,0 +1,21 @@
+{
+  "scenarios": "./scenarios",
+  "agents": [
+    {
+      "agent": "codex",
+      "flags": {
+        "full-auto": true,
+        "skip-git-repo-check": true
+      }
+    }
+  ],
+  "skills": ["./.."],
+  "settings": {
+    "concurrency": 1,
+    "limits": {
+      "scenario": {
+        "time_minutes": 10
+      }
+    }
+  }
+}

skills/pr-writer/evals/scenarios/concise-docs-pr.json

@@ -0,0 +1,52 @@
+{
+  "name": "Concise docs PR",
+  "prompt": "Use pr-writer to draft only a PR title and body for the current uncommitted docs-only skill change. Do not create or update a GitHub PR.",
+  "judge": [
+    {
+      "check": "The title uses an allowed conventional commit type, preferably docs, and accurately describes the docs-only skill guidance change.",
+      "weight": 0.3
+    },
+    {
+      "check": "The body is concise reviewer-facing prose that explains the behavior impact of the guidance without file-by-file narration.",
+      "weight": 0.3
+    },
+    {
+      "check": "The body does not include default template headings such as Summary or Test Plan.",
+      "weight": 0.2
+    },
+    {
+      "check": "The body does not include validation command dumps or agent/tool attribution unless tied to reviewer risk.",
+      "weight": 0.2
+    }
+  ],
+  "setup": [
+    {
+      "action": "run_script",
+      "command": "rsync -a --exclude .git --exclude .axis --exclude .codex --exclude .agents \"$AXIS_CONFIG_DIR/../../..\"/ ."
+    },
+    {
+      "action": "run_script",
+      "command": "git init"
+    },
+    {
+      "action": "run_script",
+      "command": "git config user.email pr-writer-eval@example.invalid"
+    },
+    {
+      "action": "run_script",
+      "command": "git config user.name 'PR Writer Eval'"
+    },
+    {
+      "action": "run_script",
+      "command": "git add ."
+    },
+    {
+      "action": "run_script",
+      "command": "git commit -m baseline"
+    },
+    {
+      "action": "run_script",
+      "command": "printf '%s\n' '' '## Concision Guidance' '' 'For documentation-only skill changes, keep generated PR bodies to one short paragraph unless the diff changes user-visible behavior, contracts, or migration risk.' >> skills/pr-writer/SKILL.md"
+    }
+  ]
+}

skills/skill-writer/EVAL.md

@@ -0,0 +1,154 @@
+# Skill Writer Evaluation
+
+Use this file to evaluate changes to `skill-writer` itself. This is a maintainer eval, not a runtime reference: do not link it from `SKILL.md`.
+
+## Framework Choice
+
+Use AXIS as the skill-eval harness.
+
+| Requirement | AXIS Fit |
+|-------------|----------|
+| open source | `@netlify/axis` is MIT licensed |
+| must use Codex | AXIS has a built-in `codex` adapter |
+| must use Codex harness | the adapter runs `codex exec --json` |
+| isolated workspaces | each scenario runs in a temp workspace with isolated `HOME` and `CODEX_HOME` |
+| LLM-as-judge | scenario `judge` checks grade the full transcript and workspace |
+| debugging | reports include transcripts, scores, artifacts, and raw agent output |
+| regressions | baselines support score comparison and CI gates |
+
+Do not add custom `run-codex.*` scripts. Add a small AXIS scenario, fixture, or artifact pattern instead.
+
+## Quick Run
+
+Run the working set:
+
+```bash
+npx @netlify/axis run --config skills/skill-writer/evals/axis.config.json
+```
+
+Run one case:
+
+```bash
+npx @netlify/axis run \
+  --config skills/skill-writer/evals/axis.config.json \
+  --scenario small-inline-workflow
+```
+
+Inspect results:
+
+```bash
+npx @netlify/axis reports latest --config skills/skill-writer/evals/axis.config.json
+npx @netlify/axis reports latest --config skills/skill-writer/evals/axis.config.json --html
+```
+
+Compare to a stored baseline:
+
+```bash
+npx @netlify/axis run \
+  --config skills/skill-writer/evals/axis.config.json \
+  --compare-baseline
+```
+
+Set or refresh a baseline only after reviewing the report:
+
+```bash
+npx @netlify/axis baseline set --config skills/skill-writer/evals/axis.config.json
+```
+
+## Evaluation Principles
+
+Use these principles when adding or changing `skill-writer` evals:
+
+1. Start with 2-3 realistic cases before expanding the suite.
+2. Compare candidate output against a baseline instead of trusting one-off scores.
+3. Write observable judge checks: files created, forbidden files absent, validation mentioned, generated instructions concise.
+4. Use LLM-as-judge for open-ended quality: concision, routed structure, source coverage, progressive disclosure, and overfitting risk.
+5. Require concrete evidence in judge checks and manual review: paths, artifacts, diffs, transcript snippets, or validation output.
+6. Track timing, token use, and interaction waste from AXIS reports.
+7. Review transcripts when a run fails or is slow; wasted steps usually point to bloated or ambiguous skill instructions.
+8. Use blind old-vs-new comparison for qualitative regressions when scores are close.
+
+## Case Set
+
+AXIS scenarios live in `evals/scenarios/`.
+
+| Case | Purpose |
+|------|---------|
+| `small-inline-workflow` | catches over-splitting and invented source requirements |
+| `reference-backed-integration` | catches bloated `SKILL.md` and hidden optional depth |
+| `iteration-from-bad-output` | catches failure to improve from concrete negative evidence |
+
+Add provider-specific cases only when changed rules affect provider-specific mechanics.
+
+## What To Test
+
+| Layer | Purpose | Evidence |
+|-------|---------|----------|
+| trigger | `skill-writer` is invoked for intended requests and skipped for unrelated requests | AXIS transcript and skill-use behavior |
+| structure | generated files obey Agent Skills format and repo registration rules | artifacts, diff, final message, validator transcript |
+| quality | generated runtime instructions are concise, routed, sourced, and actionable | AXIS judge checks and manual artifact review |
+| regression | candidate output is better than or equal to the baseline on held-out cases | AXIS baseline comparison |
+
+## AXIS Setup
+
+`evals/axis.config.json` configures:
+
+1. `agents: [{"agent": "codex"}]` so Codex is the agent under test.
+2. `skills: ["./.."]` so AXIS installs this local `skill-writer` skill for the run.
+3. scenario limits so a stuck generation fails quickly enough to debug.
+
+Each scenario configures artifacts for generated skill files, registration files, and validator output when present.
+
+Each scenario copies the repository into AXIS's isolated workspace before Codex starts. That gives `skill-writer` the same repo conventions it sees in normal use without polluting this checkout.
+
+## Rubric
+
+Score each dimension `pass`, `warn`, or `fail` during manual review. AXIS supplies the transcript, artifacts, and judge scores; this rubric decides whether to adopt the change.
+
+| Dimension | Pass | Fail |
+|-----------|------|------|
+| trigger precision | description fits should-trigger and should-not-trigger requests | overbroad or misses obvious requests |
+| artifact minimality | every file has a clear runtime, validation, source, or maintenance role | extra docs, duplicate refs, or catch-all files |
+| runtime concision | `SKILL.md` is a router when needed and added lines change decisions/actions/checks | prose repeats rationale or generic background |
+| source coverage | decisions cite enough local, official, or example evidence | important behavior depends on unsupported assumptions |
+| progressive disclosure | optional depth is routed from `SKILL.md` with clear open-when reasons | required instructions are hidden or always-loaded detail is bloated |
+| validation | structural validator runs and manual checks match the skill class/shape | validation missing, irrelevant, or treated as semantic proof |
+| portability | provider-specific mechanics are labeled and justified | provider-specific behavior leaks into portable defaults |
+
+Critical dimensions: trigger precision, artifact minimality, runtime concision, progressive disclosure, and validation.
+
+## Adoption Gate
+
+Adopt a `skill-writer` change only when:
+
+1. AXIS runs complete for the working set.
+2. no critical dimension regresses on holdout cases.
+3. at least one target dimension improves on the case that motivated the change.
+4. structural validation passes for generated skills or the transcript explains a legitimate blocker.
+5. runtime `SKILL.md` remains a router and does not link to eval files.
+6. source and maintenance notes are updated only when the skill contract changes.
+
+## Result Template
+
+```markdown
+## Eval Summary
+- Baseline:
+- Candidate:
+- AXIS report:
+- Cases run:
+- Critical regressions:
+- Decision: adopt | revise | reject
+
+## Case Results
+| Case | Score | Structure | Quality | Regression | Notes |
+|------|-------|-----------|---------|------------|-------|
+
+## Required Fixes
+- ...
+
+## Evidence
+- AXIS report:
+- generated artifacts:
+- validator output:
+- transcript notes:
+```

skills/skill-writer/SKILL.md

@@ -28,6 +28,7 @@ Follow the workflow steps in order. Load only the reference files required for t
 | iterate from positive, negative, or fix examples | `references/iteration-path.md` |
 | store persistent working and holdout examples for future revisions | `references/iteration-evidence.md` |
 | choose a response template, schema, or output contract | `references/output-contracts.md` |
+| add or update evals for a skill's generated outputs or runtime behavior | `references/skill-evals.md` |
 | troubleshoot overloaded layouts, hidden refs, or other structure failures | `references/structure-troubleshooting.md` |
 | register the skill and run final validation checks | `references/registration-validation.md` |
 
@@ -126,7 +127,8 @@ Read `references/authoring-path.md`.
    - happy-path
    - secure/robust variant
    - anti-pattern + corrected version
-12. After any skill artifact changes, run the post-change precision pass in `references/authoring-path.md` before description optimization or validation.
+12. Read `references/skill-evals.md` when the request asks for evals, regression cases, benchmark cases, or model-graded quality checks for the skill being authored.
+13. After any skill artifact changes, run the post-change precision pass in `references/authoring-path.md` before description optimization or validation.
 
 ## Step 5: Optimize description quality