feat(eval-author): reject an Insight suite whose metrics were never authored

[aleckhoury]

Tracking: ASE-780

Summary

Two ways the Insight suite could ship without the signal it exists to provide. Both turned up while running the authoring pipeline end to end against a real agent, and both are silent — the suite looks healthy in every artifact and report.

A task was never authored. The authoring agent augmented six of seven verifiers and nothing noticed. A verifier missing the Insight metric is still valid Python, so every static check passed; the gap only surfaced after a full evaluation had run, as a ragged metric key set that aborted aggregation. This snapshots each task's verifier content hash around authoring and compares them. A hash that survives byte-identical means that task never received the metric. EvalAuthorUnauthoredTasksError subclasses DatasetValidationError, so the existing repair loop feeds it back as validation feedback and names the offending tasks — turning a wasted evaluation into a few seconds.

The snapshot stays pinned to the pre-authoring baseline across repair attempts, because the question is whether a task was ever authored, not whether it changed on the latest pass.

A judge that could not run reported maximum failure. An authored LLM judge wrapped itself in except Exception: score = 0.0. In a run where the judge was misconfigured, all 50 trials recorded a valid-looking 0.0 — indistinguishable downstream from an agent that genuinely exhibited the bad behavior. The metric read as healthy while pinned to a constant, contributing a tied Pareto axis and no gradient for any consumer to climb. The authoring prompt already defines 0.0 as "complete failure" and gave the model no way to express "could not evaluate", so except: score = 0.0 was the natural thing to write. It now instructs the model to let the exception propagate and leave the metric file unwritten, which marks the trial failed and excludes it from aggregation.

Scope note

An earlier attempt at this grew ~2,030 lines into loop.py, models.py, terminator.py, and analyzer.py, splitting the suite into held-out halves and wiring the validation half into Pareto selection. That ran against the direction tests/test_plugin_boundary.py already declares — "Eval Author is meant to end up standalone, with Experimentalist depending on it and not the other way around" — so it has been handed to the Experimentalist team as #1086 instead.

The Eval Author's deliverable is a well-formed dataset of tasks, verifiers, and metrics derived from an Insight's production traces. Consumers decide how to use it: the suite ships flat, each task carries its source_trace_ref so a consumer can split by trace without leakage, and the user's train and validation datasets are still returned untouched. This PR therefore adds no Experimentalist-side code and no new rows to the boundary allowlist.

Verification

• plugins/nemo-eval-author: 64 passed, 2 skipped
• plugins/nemo-experimentalist: 594 passed (the consumer, unchanged)
• ruff check, ruff format --check, and ty check clean

New tests cover a skipped task routing into repair and recovering, a task never authored exhausting the repair budget, a task authored on an earlier attempt not being re-flagged when a later repair touches only its sibling, per-task hash independence, and hashes ignoring edits outside the verifier directory.

Test plan

• Re-run the authoring pipeline against an Insight with more traces and a working judge, and confirm the authored metric produces varying scores rather than a constant

[github-actions]

Suite	Lines Covered	Line Rate	Branch Rate
Unit Tests	30743/39237	78.3%	62.7%
Integration Tests	18062/37189	48.6%	21.0%