refactor(eval-author): simplify dataset augmentation - #1107
Merged
aleckhoury merged 20 commits intoAug 7, 2026
Conversation
Contributor
|
aleckhoury
changed the title
aleckhoury
changed the title
aleckhoury
changed the title
aleckhoury
changed the title
aleckhoury
marked this pull request as ready for review
August 6, 2026 15:01
Contributor
|
Note Reviews pausedIt looks like this branch is under active development. To avoid overwhelming you with review comments due to an influx of new commits, CodeRabbit has automatically paused this review. You can configure this behavior by changing the Use the following commands to manage reviews:
Use the checkboxes below for quick actions:
📝 WalkthroughWalkthroughEval Author now stages and validates train, validation, and Insight inputs, authors shared metrics, and returns metric metadata. Experimentalist now applies objective and regression contracts throughout optimization instead of scalar Insight evaluation and promotion. ChangesEval Author metric handoff
Experimentalist multi-objective optimization
Sequence Diagram(s)sequenceDiagram
participant Caller
participant run_eval_author
participant DatasetFactory
participant EvalAuthor
participant EvolutionaryOptimizer
participant RunReporter
Caller->>run_eval_author: provide Eval Author inputs
run_eval_author->>DatasetFactory: parse staged datasets
run_eval_author->>EvalAuthor: pass train, validation, and Insight datasets
EvalAuthor-->>Caller: return authored metrics and staged datasets
EvolutionaryOptimizer->>EvolutionaryOptimizer: build objective and regression contracts
EvolutionaryOptimizer->>RunReporter: report metric mappings and objective scores
Possibly related PRs
Suggested reviewers: 🚥 Pre-merge checks | ✅ 4 | ❌ 1❌ Failed checks (1 warning)
✅ Passed checks (4 passed)
✨ Finishing Touches 💡 2📝 Generate docstrings 💡
⚔️ Resolve merge conflicts 💡
🧪 Generate unit tests (beta)
Comment |
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
Contributor
There was a problem hiding this comment.
Actionable comments posted: 1
🧹 Nitpick comments (2)
plugins/nemo-eval-author/src/nemo_eval_author_plugin/eval_author/materialization.py (1)
84-92: 📐 Maintainability & Code Quality | 🔵 Trivial | ⚡ Quick winInclude URI resolution in the accumulated-failure path.
local_path_from_uriruns outside thetry. A non-local task URI raises immediately and discards all failures collected so far. Move it inside the guarded block so the caller sees the full report.♻️ Proposed change
- task_dir = local_path_from_uri( - task.uri, - context=f"{dataset_name} dataset task {task.id!r}", - ).resolve() try: + task_dir = local_path_from_uri( + task.uri, + context=f"{dataset_name} dataset task {task.id!r}", + ).resolve() verifier_dir = _verifier_dir(task_dir) except (FileNotFoundError, ValueError, tomllib.TOMLDecodeError) as exc: failures.append(f"{dataset_name}/{task.id}: {exc}") continue🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@plugins/nemo-eval-author/src/nemo_eval_author_plugin/eval_author/materialization.py` around lines 84 - 92, Move the task.uri resolution and .resolve() logic into the existing try block surrounding _verifier_dir(task_dir), so local_path_from_uri failures are caught by the same exception handler. Preserve the current failures.append and continue behavior to retain accumulated failures and continue processing later tasks.plugins/nemo-eval-author/tests/test_eval_author_agent.py (1)
263-263: 📐 Maintainability & Code Quality | 🔵 Trivial | ⚡ Quick winDrop
raising=Falseforvalidate_metric_contracts.
agent.pyimports the symbol at module scope, so it exists. Withraising=False, a future rename or removal silently leaves the real validator unpatched and the test still passes.♻️ Proposed change
- monkeypatch.setattr(eval_author_module, "validate_metric_contracts", validate_metric_contracts, raising=False) + monkeypatch.setattr(eval_author_module, "validate_metric_contracts", validate_metric_contracts)🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@plugins/nemo-eval-author/tests/test_eval_author_agent.py` at line 263, Update the monkeypatch call for validate_metric_contracts in the test to remove raising=False, ensuring the patch fails if the module-level symbol is renamed or removed.
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@plugins/nemo-eval-author/src/nemo_eval_author_plugin/eval_author/models.py`:
- Around line 93-96: Update the validation in the model’s identity-checking path
around insight_suite_identity to require the exact sha256: prefix followed by 64
hexadecimal characters, while retaining the existing required/non-empty
behavior. Reject identities with an invalid prefix, incorrect length, or
non-hexadecimal characters, and add tests covering each case.
---
Nitpick comments:
In
`@plugins/nemo-eval-author/src/nemo_eval_author_plugin/eval_author/materialization.py`:
- Around line 84-92: Move the task.uri resolution and .resolve() logic into the
existing try block surrounding _verifier_dir(task_dir), so local_path_from_uri
failures are caught by the same exception handler. Preserve the current
failures.append and continue behavior to retain accumulated failures and
continue processing later tasks.
In `@plugins/nemo-eval-author/tests/test_eval_author_agent.py`:
- Line 263: Update the monkeypatch call for validate_metric_contracts in the
test to remove raising=False, ensuring the patch fails if the module-level
symbol is renamed or removed.
🪄 Autofix
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Path: .coderabbit.yaml
Review profile: CHILL
Plan: Enterprise
Run ID: cc3bd9bb-1a36-4b6e-9482-2ee8713a6ece
📒 Files selected for processing (23)
plugins/nemo-eval-author/README.mdplugins/nemo-eval-author/src/nemo_eval_author_plugin/eval_author/README.mdplugins/nemo-eval-author/src/nemo_eval_author_plugin/eval_author/REFERENCE.mdplugins/nemo-eval-author/src/nemo_eval_author_plugin/eval_author/agent.pyplugins/nemo-eval-author/src/nemo_eval_author_plugin/eval_author/materialization.pyplugins/nemo-eval-author/src/nemo_eval_author_plugin/eval_author/models.pyplugins/nemo-eval-author/src/nemo_eval_author_plugin/eval_author/run.pyplugins/nemo-eval-author/tests/test_eval_author_agent.pyplugins/nemo-eval-author/tests/test_eval_author_materialization.pyplugins/nemo-eval-author/tests/test_eval_author_models.pyplugins/nemo-eval-author/tests/test_eval_author_repair_e2e.pyplugins/nemo-eval-author/tests/test_eval_author_run.pyplugins/nemo-eval-author/tests/test_plugin_boundary.pyplugins/nemo-experimentalist/src/nemo_experimentalist_plugin/entities.pyplugins/nemo-experimentalist/src/nemo_experimentalist_plugin/experimentalist/components/insight_promotion.pyplugins/nemo-experimentalist/src/nemo_experimentalist_plugin/experimentalist/components/loop.pyplugins/nemo-experimentalist/tests/experimentalist/test_dataset_staging_loop.pyplugins/nemo-experimentalist/tests/experimentalist/test_loop_helpers.pyplugins/nemo-experimentalist/tests/experimentalist/test_loop_insight_suite.pyplugins/nemo-experimentalist/tests/experimentalist/test_loop_reporting.pyplugins/nemo-experimentalist/tests/experimentalist/test_model_config.pyplugins/nemo-experimentalist/tests/test_experiment_mirror.pyplugins/nemo-experimentalist/tests/test_experimentalist_backend.py
💤 Files with no reviewable changes (3)
- plugins/nemo-experimentalist/tests/experimentalist/test_loop_reporting.py
- plugins/nemo-experimentalist/tests/experimentalist/test_loop_insight_suite.py
- plugins/nemo-experimentalist/src/nemo_experimentalist_plugin/experimentalist/components/insight_promotion.py
aleckhoury
commented
Aug 6, 2026
aleckhoury
commented
Aug 6, 2026
callingmedic911
approved these changes
Aug 6, 2026
aleckhoury
enabled auto-merge
August 6, 2026 21:34
Contributor
There was a problem hiding this comment.
Actionable comments posted: 5
🧹 Nitpick comments (3)
plugins/nemo-experimentalist/tests/experimentalist/test_reporting.py (1)
51-72: 📐 Maintainability & Code Quality | 🔵 Trivial | ⚡ Quick winAdd coverage for the decline marker and the
n/apath.Every case here renders
▲. Two branches incandidate_evaluatedstay untested: the▼marker when an objective moves against its direction, andf"{target.name} n/a"when the evaluator omits an objective metric. The second branch is the one that signals a broken metric contract, so it deserves a test.🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@plugins/nemo-experimentalist/tests/experimentalist/test_reporting.py` around lines 51 - 72, The test test_candidate_evaluated_shows_all_objectives_and_direction_aware_deltas currently covers only ▲ deltas; extend coverage for candidate_evaluated to verify a directionally worsening objective renders ▼ and an omitted objective metric renders the exact “{target.name} n/a” output. Include assertions for both branches while preserving the existing objective formatting checks.plugins/nemo-experimentalist/tests/test_metric_contract.py (1)
103-118: 📐 Maintainability & Code Quality | 🔵 Trivial | ⚡ Quick winCover the name-collision and empty-key branches of
_with_insight_objective.Two branches stay untested. First, an authored Insight key that matches a configured regression or objective name: the helper drops it from
regression_by_nameto avoid the overlapValueErrorraised byvalidate_metric_contract. Second, an emptymetric_keystuple, which returns the config unchanged. Both are cheap to assert and both guard against a run-time validation failure.🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@plugins/nemo-experimentalist/tests/test_metric_contract.py` around lines 103 - 118, Add tests for _with_insight_objective covering an authored Insight key that collides with an existing objective or regression metric, asserting the resulting configuration passes validation without duplicate names and preserves the intended metric placement. Add a separate test for an empty metric_keys tuple, asserting the helper returns the configuration unchanged.plugins/nemo-experimentalist/src/nemo_experimentalist_plugin/experimentalist/components/selector.py (1)
6-6: 📐 Maintainability & Code Quality | 🔵 Trivial | ⚡ Quick winRemove
from __future__ import annotations.Use concrete annotations to follow the plugin convention. All referenced types are imported at module scope.
nooaresolves deferred annotations withtyping.get_type_hints, so this is a consistency refactor, not a schema-generation fix.🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@plugins/nemo-experimentalist/src/nemo_experimentalist_plugin/experimentalist/components/selector.py` at line 6, Remove the from __future__ import annotations statement from selector.py, leaving the existing concrete type annotations and module-level imports unchanged.Source: Coding guidelines
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@plugins/nemo-experimentalist/README.md`:
- Around line 157-172: Split the combined YAML example in the documentation into
two separate fenced YAML blocks, each with its own descriptive introductory
text. Keep the single-metric example and the multi-metric example as
independent, valid YAML documents without duplicate objective_function keys.
In
`@plugins/nemo-experimentalist/src/nemo_experimentalist_plugin/experimentalist/components/analyzer.py`:
- Around line 417-424: Update the peer-comparison flow around
`_select_divergent_pairs` and `_find_complementary_failures` to accept and apply
each metric’s direction contract before computing deltas or selecting leaders,
so minimize metrics consistently treat lower values as better. Propagate the
metric definitions from the caller through both computations and add coverage
for a mixed maximize/minimize comparison.
In
`@plugins/nemo-experimentalist/src/nemo_experimentalist_plugin/experimentalist/components/evaluator/harbor.py`:
- Around line 1205-1220: Update the task import flow around _from_task_dir to
validate the staged task before renaming over an existing destination, and
retain the backup until reloading the replacement destination succeeds. Restore
the backup and clean up staging if validation or reload fails, preserving the
original task. Add a test covering replacement with malformed task.toml.
In
`@plugins/nemo-experimentalist/src/nemo_experimentalist_plugin/experimentalist/components/loop.py`:
- Around line 1418-1442: Update the candidate filtering flow around eligible,
excluded, and ranked to handle the case where every candidate is excluded:
either fail fast with an explicit error or fall back to the original candidates
so the round cannot proceed with zero survivors. Build the exclusion list by
comparing candidate labels against the labels of eligible candidates, rather
than comparing Pydantic model instances by value.
In
`@plugins/nemo-experimentalist/src/nemo_experimentalist_plugin/experimentalist/components/terminator.py`:
- Around line 196-200: Update the call to qualitative_stop_check to pass the
resolved active_objectives and active_regressions values instead of
objective_metrics and regression_metrics, preserving the fallback behavior
established when those active lists are resolved.
---
Nitpick comments:
In
`@plugins/nemo-experimentalist/src/nemo_experimentalist_plugin/experimentalist/components/selector.py`:
- Line 6: Remove the from __future__ import annotations statement from
selector.py, leaving the existing concrete type annotations and module-level
imports unchanged.
In `@plugins/nemo-experimentalist/tests/experimentalist/test_reporting.py`:
- Around line 51-72: The test
test_candidate_evaluated_shows_all_objectives_and_direction_aware_deltas
currently covers only ▲ deltas; extend coverage for candidate_evaluated to
verify a directionally worsening objective renders ▼ and an omitted objective
metric renders the exact “{target.name} n/a” output. Include assertions for both
branches while preserving the existing objective formatting checks.
In `@plugins/nemo-experimentalist/tests/test_metric_contract.py`:
- Around line 103-118: Add tests for _with_insight_objective covering an
authored Insight key that collides with an existing objective or regression
metric, asserting the resulting configuration passes validation without
duplicate names and preserves the intended metric placement. Add a separate test
for an empty metric_keys tuple, asserting the helper returns the configuration
unchanged.
🪄 Autofix
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Path: .coderabbit.yaml
Review profile: CHILL
Plan: Enterprise
Run ID: 827baef6-02cc-454c-8a52-097573fa0d87
📒 Files selected for processing (22)
plugins/nemo-experimentalist/README.mdplugins/nemo-experimentalist/examples/tau3-nooa-agent/.env.exampleplugins/nemo-experimentalist/examples/tau3-nooa-agent/experimentalist-smoke.yamlplugins/nemo-experimentalist/src/nemo_experimentalist_plugin/config.pyplugins/nemo-experimentalist/src/nemo_experimentalist_plugin/entities.pyplugins/nemo-experimentalist/src/nemo_experimentalist_plugin/experimentalist/components/analyzer.pyplugins/nemo-experimentalist/src/nemo_experimentalist_plugin/experimentalist/components/cache.pyplugins/nemo-experimentalist/src/nemo_experimentalist_plugin/experimentalist/components/dataset_staging.pyplugins/nemo-experimentalist/src/nemo_experimentalist_plugin/experimentalist/components/evaluator/harbor.pyplugins/nemo-experimentalist/src/nemo_experimentalist_plugin/experimentalist/components/loop.pyplugins/nemo-experimentalist/src/nemo_experimentalist_plugin/experimentalist/components/proposer.pyplugins/nemo-experimentalist/src/nemo_experimentalist_plugin/experimentalist/components/selector.pyplugins/nemo-experimentalist/src/nemo_experimentalist_plugin/experimentalist/components/terminator.pyplugins/nemo-experimentalist/src/nemo_experimentalist_plugin/experimentalist/components/trace_analyzer.pyplugins/nemo-experimentalist/src/nemo_experimentalist_plugin/experimentalist/experimentalist_backend.pyplugins/nemo-experimentalist/src/nemo_experimentalist_plugin/experimentalist/reporting.pyplugins/nemo-experimentalist/tests/experimentalist/test_dataset_staging.pyplugins/nemo-experimentalist/tests/experimentalist/test_dataset_staging_loop.pyplugins/nemo-experimentalist/tests/experimentalist/test_reporting.pyplugins/nemo-experimentalist/tests/experimentalist/test_terminator.pyplugins/nemo-experimentalist/tests/test_experimentalist_analyzer.pyplugins/nemo-experimentalist/tests/test_metric_contract.py
💤 Files with no reviewable changes (1)
- plugins/nemo-experimentalist/examples/tau3-nooa-agent/experimentalist-smoke.yaml
🚧 Files skipped from review as they are similar to previous changes (1)
- plugins/nemo-experimentalist/tests/experimentalist/test_dataset_staging_loop.py
Contributor
|
Note GitHub couldn't provide a complete incremental comparison for this pull request, so CodeRabbit is performing a full review instead. This review may take a little longer. |
Contributor
There was a problem hiding this comment.
Actionable comments posted: 2
🧹 Nitpick comments (3)
plugins/nemo-eval-author/tests/test_eval_author_agent.py (1)
263-263: 📐 Maintainability & Code Quality | 🔵 Trivial | ⚡ Quick winDrop
raising=False.
agent.pyimportsvalidate_metric_contracts, so the attribute exists. Withraising=False, a future rename silently creates a dead patch and the tests keep passing against unpatched code.🔧 Proposed fix
- monkeypatch.setattr(eval_author_module, "validate_metric_contracts", validate_metric_contracts, raising=False) + monkeypatch.setattr(eval_author_module, "validate_metric_contracts", validate_metric_contracts)🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@plugins/nemo-eval-author/tests/test_eval_author_agent.py` at line 263, Update the monkeypatch in the test around validate_metric_contracts to remove raising=False, relying on the imported validate_metric_contracts attribute being present so renamed symbols fail loudly instead of silently creating an ineffective patch.plugins/nemo-experimentalist/src/nemo_experimentalist_plugin/config.py (1)
189-190: 📐 Maintainability & Code Quality | 🔵 Trivial | ⚡ Quick winReplace the string-based return annotation.
Use
typing.Selfinstead of"EvolutionaryOptimizerConfig".♻️ Proposed change
`@model_validator`(mode="after") - def validate_metric_contract(self) -> "EvolutionaryOptimizerConfig": + def validate_metric_contract(self) -> Self:Add
Selfto the typing import at line 23:-from typing import Any, Literal +from typing import Any, Literal, SelfAs per coding guidelines: "Prefer concrete type hints over string-based type hints".
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@plugins/nemo-experimentalist/src/nemo_experimentalist_plugin/config.py` around lines 189 - 190, Update the return annotation of EvolutionaryOptimizerConfig.validate_metric_contract to use typing.Self instead of the quoted class name, and add Self to the existing typing imports.Source: Coding guidelines
plugins/nemo-eval-author/tests/test_eval_author_repair_e2e.py (1)
555-566: 📐 Maintainability & Code Quality | 🔵 Trivial | ⚡ Quick winDo not require exactly one authored metric key.
author_insight_metricsmay author several keys. Line 557 fails the canary when the model authors two. Compare every authored key instead.♻️ Proposed change
- assert len(violating_metric_names) == 1 - assert compliant_metric_names == violating_metric_names - metric_name = next(iter(violating_metric_names)) - violating_score = violating_payload[metric_name] - compliant_score = compliant_payload[metric_name] - assert isinstance(violating_score, int | float) and not isinstance(violating_score, bool) - assert isinstance(compliant_score, int | float) and not isinstance(compliant_score, bool) - assert 0.0 <= violating_score <= 1.0 - assert 0.0 <= compliant_score <= 1.0 - assert compliant_score > violating_score + assert violating_metric_names + assert compliant_metric_names == violating_metric_names + for metric_name in violating_metric_names: + violating_score = violating_payload[metric_name] + compliant_score = compliant_payload[metric_name] + assert isinstance(violating_score, int | float) and not isinstance(violating_score, bool) + assert isinstance(compliant_score, int | float) and not isinstance(compliant_score, bool) + assert 0.0 <= violating_score <= 1.0 + assert 0.0 <= compliant_score <= 1.0 + assert any( + compliant_payload[name] > violating_payload[name] for name in violating_metric_names + )🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@plugins/nemo-eval-author/tests/test_eval_author_repair_e2e.py` around lines 555 - 566, Update the assertions in the evaluation comparison around violating_metric_names and compliant_metric_names so multiple authored metric keys are supported instead of requiring exactly one. Compare every authored key present in both payloads, validate each violating_score and compliant_score as numeric values in [0, 1], and require compliant_score > violating_score for every key while preserving reward exclusion.
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@plugins/nemo-eval-author/src/nemo_eval_author_plugin/eval_author/models.py`:
- Line 96: Update EvalAuthorResult’s metric_keys validation around
_validate_metric_keys to assign and return the normalized tuple rather than
discarding it. Implement this as a field validator that skips the empty default,
preserving empty defaults while stripping whitespace from supplied keys
consistently with MetricAuthoringResult.
In `@plugins/nemo-experimentalist/src/nemo_experimentalist_plugin/config.py`:
- Around line 83-122: Update satisfies_regression_constraints and
is_eligible_for_metric_contract to handle a baseline_metrics mapping that lacks
declared regression targets before comparing candidate values. Choose and
implement an explicit failure or documented fallback policy so missing round-0
regression metrics cannot silently reject every candidate, while preserving the
existing constraint comparisons when baseline values are present.
---
Nitpick comments:
In `@plugins/nemo-eval-author/tests/test_eval_author_agent.py`:
- Line 263: Update the monkeypatch in the test around validate_metric_contracts
to remove raising=False, relying on the imported validate_metric_contracts
attribute being present so renamed symbols fail loudly instead of silently
creating an ineffective patch.
In `@plugins/nemo-eval-author/tests/test_eval_author_repair_e2e.py`:
- Around line 555-566: Update the assertions in the evaluation comparison around
violating_metric_names and compliant_metric_names so multiple authored metric
keys are supported instead of requiring exactly one. Compare every authored key
present in both payloads, validate each violating_score and compliant_score as
numeric values in [0, 1], and require compliant_score > violating_score for
every key while preserving reward exclusion.
In `@plugins/nemo-experimentalist/src/nemo_experimentalist_plugin/config.py`:
- Around line 189-190: Update the return annotation of
EvolutionaryOptimizerConfig.validate_metric_contract to use typing.Self instead
of the quoted class name, and add Self to the existing typing imports.
🪄 Autofix
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Path: .coderabbit.yaml
Review profile: CHILL
Plan: Enterprise
Run ID: f7a29ec0-5340-4368-8879-387dc0a453d9
📒 Files selected for processing (42)
plugins/nemo-eval-author/README.mdplugins/nemo-eval-author/src/nemo_eval_author_plugin/eval_author/README.mdplugins/nemo-eval-author/src/nemo_eval_author_plugin/eval_author/REFERENCE.mdplugins/nemo-eval-author/src/nemo_eval_author_plugin/eval_author/agent.pyplugins/nemo-eval-author/src/nemo_eval_author_plugin/eval_author/materialization.pyplugins/nemo-eval-author/src/nemo_eval_author_plugin/eval_author/models.pyplugins/nemo-eval-author/src/nemo_eval_author_plugin/eval_author/run.pyplugins/nemo-eval-author/tests/test_eval_author_agent.pyplugins/nemo-eval-author/tests/test_eval_author_materialization.pyplugins/nemo-eval-author/tests/test_eval_author_models.pyplugins/nemo-eval-author/tests/test_eval_author_repair_e2e.pyplugins/nemo-eval-author/tests/test_eval_author_run.pyplugins/nemo-eval-author/tests/test_plugin_boundary.pyplugins/nemo-experimentalist/README.mdplugins/nemo-experimentalist/examples/tau3-nooa-agent/.env.exampleplugins/nemo-experimentalist/examples/tau3-nooa-agent/experimentalist-smoke.yamlplugins/nemo-experimentalist/src/nemo_experimentalist_plugin/config.pyplugins/nemo-experimentalist/src/nemo_experimentalist_plugin/entities.pyplugins/nemo-experimentalist/src/nemo_experimentalist_plugin/experimentalist/components/analyzer.pyplugins/nemo-experimentalist/src/nemo_experimentalist_plugin/experimentalist/components/cache.pyplugins/nemo-experimentalist/src/nemo_experimentalist_plugin/experimentalist/components/dataset_staging.pyplugins/nemo-experimentalist/src/nemo_experimentalist_plugin/experimentalist/components/evaluator/harbor.pyplugins/nemo-experimentalist/src/nemo_experimentalist_plugin/experimentalist/components/insight_promotion.pyplugins/nemo-experimentalist/src/nemo_experimentalist_plugin/experimentalist/components/loop.pyplugins/nemo-experimentalist/src/nemo_experimentalist_plugin/experimentalist/components/proposer.pyplugins/nemo-experimentalist/src/nemo_experimentalist_plugin/experimentalist/components/selector.pyplugins/nemo-experimentalist/src/nemo_experimentalist_plugin/experimentalist/components/terminator.pyplugins/nemo-experimentalist/src/nemo_experimentalist_plugin/experimentalist/components/trace_analyzer.pyplugins/nemo-experimentalist/src/nemo_experimentalist_plugin/experimentalist/experimentalist_backend.pyplugins/nemo-experimentalist/src/nemo_experimentalist_plugin/experimentalist/reporting.pyplugins/nemo-experimentalist/tests/experimentalist/test_dataset_staging.pyplugins/nemo-experimentalist/tests/experimentalist/test_dataset_staging_loop.pyplugins/nemo-experimentalist/tests/experimentalist/test_loop_helpers.pyplugins/nemo-experimentalist/tests/experimentalist/test_loop_insight_suite.pyplugins/nemo-experimentalist/tests/experimentalist/test_loop_reporting.pyplugins/nemo-experimentalist/tests/experimentalist/test_model_config.pyplugins/nemo-experimentalist/tests/experimentalist/test_reporting.pyplugins/nemo-experimentalist/tests/experimentalist/test_terminator.pyplugins/nemo-experimentalist/tests/test_experiment_mirror.pyplugins/nemo-experimentalist/tests/test_experimentalist_analyzer.pyplugins/nemo-experimentalist/tests/test_experimentalist_backend.pyplugins/nemo-experimentalist/tests/test_metric_contract.py
💤 Files with no reviewable changes (4)
- plugins/nemo-experimentalist/examples/tau3-nooa-agent/experimentalist-smoke.yaml
- plugins/nemo-experimentalist/tests/experimentalist/test_loop_reporting.py
- plugins/nemo-experimentalist/tests/experimentalist/test_loop_insight_suite.py
- plugins/nemo-experimentalist/src/nemo_experimentalist_plugin/experimentalist/components/insight_promotion.py
🚧 Files skipped from review as they are similar to previous changes (30)
- plugins/nemo-eval-author/tests/test_plugin_boundary.py
- plugins/nemo-experimentalist/README.md
- plugins/nemo-eval-author/README.md
- plugins/nemo-eval-author/src/nemo_eval_author_plugin/eval_author/README.md
- plugins/nemo-experimentalist/tests/experimentalist/test_model_config.py
- plugins/nemo-experimentalist/tests/experimentalist/test_terminator.py
- plugins/nemo-experimentalist/tests/test_experimentalist_backend.py
- plugins/nemo-experimentalist/src/nemo_experimentalist_plugin/experimentalist/experimentalist_backend.py
- plugins/nemo-experimentalist/tests/test_experiment_mirror.py
- plugins/nemo-experimentalist/tests/experimentalist/test_reporting.py
- plugins/nemo-eval-author/src/nemo_eval_author_plugin/eval_author/REFERENCE.md
- plugins/nemo-experimentalist/src/nemo_experimentalist_plugin/experimentalist/components/dataset_staging.py
- plugins/nemo-experimentalist/tests/test_experimentalist_analyzer.py
- plugins/nemo-eval-author/src/nemo_eval_author_plugin/eval_author/materialization.py
- plugins/nemo-eval-author/tests/test_eval_author_models.py
- plugins/nemo-experimentalist/src/nemo_experimentalist_plugin/experimentalist/components/evaluator/harbor.py
- plugins/nemo-experimentalist/tests/test_metric_contract.py
- plugins/nemo-experimentalist/src/nemo_experimentalist_plugin/experimentalist/components/proposer.py
- plugins/nemo-eval-author/src/nemo_eval_author_plugin/eval_author/run.py
- plugins/nemo-experimentalist/src/nemo_experimentalist_plugin/entities.py
- plugins/nemo-experimentalist/src/nemo_experimentalist_plugin/experimentalist/components/cache.py
- plugins/nemo-experimentalist/src/nemo_experimentalist_plugin/experimentalist/components/trace_analyzer.py
- plugins/nemo-experimentalist/tests/experimentalist/test_dataset_staging.py
- plugins/nemo-experimentalist/src/nemo_experimentalist_plugin/experimentalist/reporting.py
- plugins/nemo-experimentalist/src/nemo_experimentalist_plugin/experimentalist/components/terminator.py
- plugins/nemo-eval-author/src/nemo_eval_author_plugin/eval_author/agent.py
- plugins/nemo-experimentalist/tests/experimentalist/test_dataset_staging_loop.py
- plugins/nemo-experimentalist/src/nemo_experimentalist_plugin/experimentalist/components/selector.py
- plugins/nemo-experimentalist/src/nemo_experimentalist_plugin/experimentalist/components/analyzer.py
- plugins/nemo-experimentalist/src/nemo_experimentalist_plugin/experimentalist/components/loop.py
gaiadilorenzo
force-pushed
the
ASE-780-eval-author-portable-verifier-patch/akhoury
branch
from
August 7, 2026 09:49
4fe22f6 to
1a657e7
Compare
8 tasks
Introduce a CLI-safe request/result boundary and immutable reference inventory so verifier authoring can be consumed independently of Experimentalist datasets. Signed-off-by: Alec Khoury <akhoury@nvidia.com>
Remove inferred snapshot and inventory machinery so metric authoring explicitly produces the reusable files alongside the generated tasks. Signed-off-by: Alec Khoury <akhoury@nvidia.com>
Restore per-task verifier authoring across staged train and validation datasets, and apply the same metrics to the generated Insight task set. Signed-off-by: Alec Khoury <akhoury@nvidia.com>
Preserve the original augmentation contract so Experimentalist can continue using the staged Dataset objects without an artifact rehydration layer. Signed-off-by: Alec Khoury <akhoury@nvidia.com>
Leave the authored Insight suite at the Eval Author handoff boundary so the Experimentalist team can define splitting, evaluation, and reward integration without legacy channel behavior. Signed-off-by: Alec Khoury <akhoury@nvidia.com>
Remove stale claims that train and validation remain unchanged or that Experimentalist immediately evaluates the authored Insight suite. Signed-off-by: Alec Khoury <akhoury@nvidia.com>
Signed-off-by: Alec Khoury <akhoury@nvidia.com>
Signed-off-by: Gaia Di Lorenzo <gdilorenzo@ethz.ch>
Signed-off-by: Gaia Di Lorenzo <gdilorenzo@ethz.ch>
Signed-off-by: Gaia Di Lorenzo <gdilorenzo@ethz.ch>
Signed-off-by: Gaia Di Lorenzo <gdilorenzo@ethz.ch>
Signed-off-by: Gaia Di Lorenzo <gdilorenzo@ethz.ch>
Signed-off-by: Gaia Di Lorenzo <gdilorenzo@ethz.ch>
Signed-off-by: Gaia Di Lorenzo <gdilorenzo@ethz.ch>
Signed-off-by: Gaia Di Lorenzo <gdilorenzo@ethz.ch>
Store normalized metric_keys on EvalAuthorResult, accumulate URI failures in contract validation, and harden related unit/canary asserts. Signed-off-by: Alec Khoury <akhoury@nvidia.com>
Signed-off-by: Gaia Di Lorenzo <75574163+gaiadilorenzo@users.noreply.github.com>
Signed-off-by: Gaia Di Lorenzo <gdilorenzo@ethz.ch>
gaiadilorenzo
force-pushed
the
ASE-780-eval-author-portable-verifier-patch/akhoury
branch
from
August 7, 2026 14:36
1a657e7 to
d125c90
Compare
…ortable-verifier-patch/akhoury Signed-off-by: Alec Khoury <akhoury@nvidia.com> # Conflicts: # plugins/nemo-experimentalist/src/nemo_experimentalist_plugin/experimentalist/components/proposer.py
…ortable-verifier-patch/akhoury Signed-off-by: Alec Khoury <akhoury@nvidia.com>
aleckhoury
deleted the
ASE-780-eval-author-portable-verifier-patch/akhoury
branch
August 7, 2026 16:08
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Refactors Eval Author’s portable metric handoff and moves Experimentalist to metric-contract-driven optimization. Regression metrics are selector context rather than a hard eligibility gate, while candidates must still report every Pareto objective.
Changes
Type of Change
Quality Gates
Verification
Signed-off-by:traileruv run pre-commit run -apasses — blocked by 38 existing type-check diagnostics from rebasedmainfiles outside this PR.Targeted validation:
.venv/bin/pytest plugins/nemo-experimentalist/tests/test_metric_contract.py plugins/nemo-experimentalist/tests/experimentalist/test_dataset_staging.py plugins/nemo-experimentalist/tests/test_experimentalist_analyzer.py plugins/nemo-experimentalist/tests/experimentalist/test_terminator.py -v— 32 passed..venv/bin/pytest plugins/nemo-experimentalist/tests/test_experimentalist_backend.py -v— 19 passed.uv run generate-config-docs && git diff --exit-code docs/set-up/config-reference.mdx— passed.