feat(experimentalist): add a fast, diagnostic smoke agent fixture - #1089
Open
schuellc-nvidia wants to merge 4 commits into
Open
feat(experimentalist): add a fast, diagnostic smoke agent fixture#1089schuellc-nvidia wants to merge 4 commits into
schuellc-nvidia wants to merge 4 commits into
Conversation
Contributor
📝 WalkthroughWalkthroughSummaryThis PR adds the deterministic ChangesSmoke-agent fixture
Experimentalist resilience
Sequence Diagram(s)sequenceDiagram
participant Harbor
participant WrappedAgent
participant main.py
participant ReportAgent
participant Verifier
Harbor->>WrappedAgent: upload selected agent files
WrappedAgent->>main.py: run instruction and session ID
main.py->>ReportAgent: solve instruction
ReportAgent-->>main.py: return formatted answer
main.py-->>WrappedAgent: write output.txt
WrappedAgent-->>Harbor: return process metadata
Verifier->>Verifier: normalize and compare output
Verifier-->>Harbor: write reward.json
Possibly related PRs
Suggested reviewers: 🚥 Pre-merge checks | ✅ 5✅ Passed checks (5 passed)
✨ Finishing Touches📝 Generate docstrings
🧪 Generate unit tests (beta)
Comment |
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
Contributor
There was a problem hiding this comment.
Actionable comments posted: 20
Caution
Some comments are outside the diff and can’t be posted inline due to platform limitations.
⚠️ Outside diff range comments (1)
plugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g4-dispatch-order/train/count-research/instruction.md (1)
1-10: 📐 Maintainability & Code Quality | 🟠 Major | 🏗️ Heavy liftAdd required SPDX metadata to all fixture files.
These files lack both required SPDX identifiers. Update the raw-fixture format and verifier handling so metadata does not become part of the expected agent output.
plugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g4-dispatch-order/train/count-research/instruction.md#L1-L10: add SPDX metadata.plugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g4-dispatch-order/train/count-research/tests/expected.txt#L1-L1: make the expected-output format SPDX-compatible.plugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g4-dispatch-order/train/lookup-ada/instruction.md#L1-L10: add SPDX metadata.plugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g5-edge-cases/train/missing-person/tests/expected.txt#L1-L1: make the expected-output format SPDX-compatible.plugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g5-edge-cases/validation/lookup-grace/instruction.md#L1-L10: add SPDX metadata.plugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g5-edge-cases/validation/lookup-grace/tests/expected.txt#L1-L1: make the expected-output format SPDX-compatible.plugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g5-edge-cases/validation/missing-person-hours/instruction.md#L1-L10: add SPDX metadata.As per coding guidelines, “Every file must contain the NVIDIA SPDX copyright header and Apache-2.0 license identifier.”
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@plugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g4-dispatch-order/train/count-research/instruction.md` around lines 1 - 10, Add the standard NVIDIA SPDX copyright header and Apache-2.0 license identifier to every listed instruction.md and expected.txt fixture, updating the raw-fixture format and verifier handling so these metadata lines are excluded from expected agent output. Apply the changes to all seven specified files, preserving each instruction and expected result content.Source: Coding guidelines
🧹 Nitpick comments (1)
plugins/nemo-experimentalist/examples/smoke-agent/README.md (1)
15-140: 📐 Maintainability & Code Quality | 🔵 Trivial | 🏗️ Heavy liftSplit the documentation by Diataxis type.
This README combines explanation, reference, how-to steps, and timings. Keep this page as a HOW-TO, then move design, layout, and timings to linked EXPLANATION or REFERENCE pages.
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@plugins/nemo-experimentalist/examples/smoke-agent/README.md` around lines 15 - 140, The smoke-agent README currently mixes how-to guidance with explanation/reference content and timings. Keep the README focused on the run/check workflow, and move the Design, Layout, Scenarios, and Timings sections into separate linked pages classified as EXPLANATION or REFERENCE. Update the existing section anchors and links so the README still points to the relocated content using the visible README sections as the split points.Source: Coding guidelines
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@plugins/nemo-experimentalist/docs/smoke-agent-weaknesses.md`:
- Around line 4-16: Update the documentation page’s front matter or opening
metadata to identify it as an EXPLANATION, then add a prerequisites section
before the existing overview. Append a Next Steps section at the end linking to
the smoke-agent implementation and its guard tests, preserving the current
weakness documentation.
In
`@plugins/nemo-experimentalist/examples/smoke-agent/configs/generalization.yaml`:
- Around line 17-25: Select a distinct held-out dataset for the generalization
scenario in the configuration around max_train_batch_tasks, using a separate
profile or supported dataset/config override rather than inheriting
optimizer.yaml’s g1-aggregation settings. Ensure the documented command resolves
to this generalization dataset instead of the repair scenario, and update the
corresponding README scenario description to match.
In
`@plugins/nemo-experimentalist/examples/smoke-agent/dataset/_shared/Dockerfile`:
- Around line 16-17: Replace the pip-based installation in the Dockerfile
dependency layer with the project’s uv-based dependency-install flow, preserving
the existing nooa[tracing] Git dependency and pinned revision. Do not introduce
or retain pip usage.
In `@plugins/nemo-experimentalist/examples/smoke-agent/dataset/_shared/test.sh`:
- Around line 31-46: Reject symlinked output artifacts before scoring in both
plugins/nemo-experimentalist/examples/smoke-agent/dataset/_shared/test.sh lines
31-46 and
plugins/nemo-experimentalist/examples/smoke-agent/dataset/task-template/tests/test.sh
lines 31-46. Update the scoring flow around the OUTPUT file check so validation
uses an isolated finalized artifact or race-safe no-follow access, preventing [
-f ], head, sed, and cmp from following an agent-controlled symlink; preserve
the existing shape and byte-for-byte comparison behavior for regular files.
In
`@plugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g1-aggregation/train/lookup-ada/instruction.md`:
- Around line 1-9: Every listed instruction.md and tests/expected.txt file must
include the NVIDIA SPDX copyright header and Apache-2.0 license identifier. Add
headers to the six specified files while preserving each instruction’s task
contract and expected output; update the expected-output parsing/verifier as
needed so the added metadata is ignored and comparisons remain unchanged.
Affected files:
plugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g1-aggregation/train/lookup-ada/instruction.md
(lines 1-9), train/lookup-ada/tests/expected.txt (line 1),
validation/total-hours-analysts/instruction.md (lines 1-9),
validation/total-hours-analysts/tests/expected.txt (line 1),
validation/total-hours-ops/instruction.md (lines 1-9), and
validation/total-hours-ops/tests/expected.txt (line 1).
In
`@plugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g1-aggregation/train/lookup-ada/tests/test.sh`:
- Around line 35-36: Update the shape_ok assignment checks in all three listed
test.sh files at lines 35-36 to require at least one non-whitespace character
after the equals sign, so entries such as total= or dept= are rejected while
populated values remain accepted.
In
`@plugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g1-aggregation/train/total-hours-engineers/instruction.md`:
- Around line 1-9: Resolve SPDX compliance across all six listed fixture files:
add the NVIDIA copyright header and Apache-2.0 SPDX identifier to the three
instruction.md files, while preserving their task content. For each listed
expected.txt file, replace the raw byte-exact format with the verifier’s
supported header-bearing fixture format, or add and enforce an explicit
exemption without inserting header text into the existing payload.
In
`@plugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g2-name-patterns/train/lookup-ada/instruction.md`:
- Around line 1-9: Resolve SPDX compliance through the approved fixture-policy
mechanism rather than modifying fixture contents: apply the prompt-fixture
exception to
plugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g2-name-patterns/train/lookup-ada/instruction.md
lines 1-9,
plugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g3-long-inputs/train/preamble-dept/instruction.md
lines 1-9, and
plugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g3-long-inputs/train/preamble-role/instruction.md
lines 1-9. Apply the approved literal-artifact exception to
plugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g2-name-patterns/train/lookup-ada/tests/expected.txt
line 1,
plugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g3-long-inputs/train/plain-dept/tests/expected.txt
line 1, and
plugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g3-long-inputs/train/preamble-dept/tests/expected.txt
line 1; preserve all prompt and exact expected-output text unchanged while
ensuring the policy recognizes the NVIDIA copyright and Apache-2.0 licensing
requirement.
In
`@plugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g2-name-patterns/train/lookup-obrien/instruction.md`:
- Line 1: Resolve missing SPDX metadata across the listed fixtures: add the
NVIDIA copyright and Apache-2.0 identifiers to
plugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g2-name-patterns/train/lookup-obrien/instruction.md#L1-L1,
plugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g2-name-patterns/train/lookup-zoe/instruction.md#L1-L1,
plugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g3-long-inputs/validation/preamble-hours/instruction.md#L1-L1,
and
plugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g3-long-inputs/validation/preamble-long-dept/instruction.md#L1-L1;
apply the approved non-semantic exact-output-fixture metadata mechanism to
plugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g2-name-patterns/train/lookup-obrien/tests/expected.txt#L1-L1,
plugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g3-long-inputs/train/preamble-role/tests/expected.txt#L1-L1,
and
plugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g3-long-inputs/validation/preamble-hours/tests/expected.txt#L1-L1
without inserting literal headers or changing verifier bytes.
In
`@plugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g2-name-patterns/train/lookup-zoe/tests/expected.txt`:
- Line 1: Add the required NVIDIA SPDX copyright and Apache-2.0 license metadata
to
plugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g2-name-patterns/train/lookup-zoe/tests/expected.txt:1-1,
plugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g3-long-inputs/validation/preamble-long-dept/tests/expected.txt:1-1,
and
plugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g3-long-inputs/validation/trailing-prose/tests/expected.txt:1-1
without altering the compared payload; update the exact-output fixture format or
verifier so these headers are ignored during comparison. Add the same required
SPDX header to
plugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g2-name-patterns/validation/lookup-ann-marie/instruction.md:1-9
and
plugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g3-long-inputs/validation/trailing-prose/instruction.md:1-9.
In
`@plugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g2-name-patterns/validation/lookup-ann-marie/tests/expected.txt`:
- Line 1: Document an approved SPDX licensing exception or metadata rule for all
affected content-sensitive dataset fixtures, without modifying their bytes or
task behavior. Apply this to
plugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g2-name-patterns/validation/lookup-ann-marie/tests/expected.txt:1,
plugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g2-name-patterns/validation/lookup-grace/instruction.md:1-9
and tests/expected.txt:1,
plugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g2-name-patterns/validation/lookup-role-obrien/instruction.md:1-9
and tests/expected.txt:1, and
plugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g3-long-inputs/train/plain-dept/instruction.md:1-9;
do not prepend SPDX headers or otherwise alter fixture contents.
In
`@plugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g4-dispatch-order/train/count-ops/instruction.md`:
- Line 1: Add the standard NVIDIA SPDX copyright and Apache-2.0 license
identifiers to the Markdown instruction files at
plugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g4-dispatch-order/train/count-ops/instruction.md:1-1,
plugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g5-edge-cases/train/lookup-ada/instruction.md:1-1,
and
plugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g5-edge-cases/train/missing-person/instruction.md:1-1.
Update the expected-output fixtures at
plugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g4-dispatch-order/train/count-ops/tests/expected.txt:1-1,
plugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g5-edge-cases/train/empty-role/tests/expected.txt:1-1,
and
plugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g5-edge-cases/train/lookup-ada/tests/expected.txt:1-1
to use the licensed fixture representation, and update the canonical verifier to
ignore the added metadata while preserving expected-output semantics.
In
`@plugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g4-dispatch-order/train/count-research/tests/test.sh`:
- Around line 29-49: Add symlink and same-inode rejection before the output
scoring branch in each test.sh:
plugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g4-dispatch-order/train/count-research/tests/test.sh
lines 29-49,
plugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g5-edge-cases/train/missing-person/tests/test.sh
lines 29-49, and
plugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g5-edge-cases/validation/lookup-grace/tests/test.sh
lines 29-49. Before the existing [ -f "$OUTPUT" ] check, exit successfully after
reporting a symbolic-link OUTPUT or when stat confirms OUTPUT and EXPECTED_FILE
share an inode, while guarding inode comparison on EXPECTED_FILE readability.
In
`@plugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g4-dispatch-order/train/lookup-ada/tests/expected.txt`:
- Line 1: Update the affected fixtures and instructions to include the required
NVIDIA SPDX/Apache-2.0 header while keeping the expected payloads unchanged, and
adjust each listed verifier to strip or ignore license-header lines before
comparing output. In the g4-dispatch-order lookup-ada test flow, update
plugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g4-dispatch-order/train/lookup-ada/tests/expected.txt
and the matching tests/test.sh comparison logic; do the same for the
g5-edge-cases missing-person-hours and missing-person-role test fixtures and
test.sh scripts. Also add Markdown-compatible SPDX identifiers to the two
instruction.md files so the task text remains valid while satisfying the
file-header requirement.
In `@plugins/nemo-experimentalist/examples/smoke-agent/README.md`:
- Around line 61-79: Update the README commands around scripts/build_image.py
and the scenario execution to use the required uv environment: invoke the build
script through uv run, and remove the UV_PROJECT_ENVIRONMENT override or set it
to the repository root .venv. Keep the existing package and plugin arguments
unchanged.
In
`@plugins/nemo-experimentalist/tests/experimentalist/test_smoke_agent_assets.py`:
- Around line 41-74: Strengthen test_verifier_emits_exactly_the_two_metric_keys
and test_verifier_keeps_its_reward_hacking_guards to validate complete
invariants rather than substring fragments: parse the verifier’s emitted metric
keys and require exactly reward and shape_ok, and require a set command
containing both -u and pipefail while still rejecting -e. Use _verifier_code to
ignore comments and preserve the existing checks for tr -d, cmp -s, and refusing
to score.
In `@plugins/nemo-experimentalist/tests/experimentalist/test_smoke_agent_gate.py`:
- Around line 105-112: Update the winner validation in the smoke test around
_winner_label() and _agent_source() so it no longer treats the presence of
“sum(” as aggregation evidence. Instead, verify a changed execution path that
sums the relevant record field for total-hours support, or replay a total-hours
input against the winning agent and assert the expected result.
- Around line 195-220: Update test_g_2_a_candidate_actually_fixed_train to
retain the best train-improving candidate’s identity, then compare that
candidate’s validation reward from _validation_aggregate against agent-0’s
validation reward. Assert the candidate does not beat the baseline while
preserving the existing requirement that its train reward is higher.
In
`@plugins/nemo-experimentalist/tests/experimentalist/test_smoke_agent_traces.py`:
- Around line 30-41: Update test_trace_carries_the_method_call_graph to select a
recorded trace containing the G4 call path, rather than assuming _traces()[0] is
suitable. Search available traces for one whose method counts include
ReportAgent.handle_list, or use a dedicated known G4 trace, while preserving the
existing solve and handle_list assertions.
In `@plugins/nemo-experimentalist/tests/experimentalist/test_smoke_agent.py`:
- Around line 106-132: Update _LEAK_SCAN_SKIP_DIRS in
test_example_directory_does_not_leak_the_weaknesses to match
_AGENT_COPY_EXCLUDE_NAMES exactly, including traces and the other
copier-excluded directories such as .git, .claude, .uv, eval-and-optimize, and
scratch; remove any exclusions not present in that copier set so the leak scan
covers precisely the files available to the Coder.
---
Outside diff comments:
In
`@plugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g4-dispatch-order/train/count-research/instruction.md`:
- Around line 1-10: Add the standard NVIDIA SPDX copyright header and Apache-2.0
license identifier to every listed instruction.md and expected.txt fixture,
updating the raw-fixture format and verifier handling so these metadata lines
are excluded from expected agent output. Apply the changes to all seven
specified files, preserving each instruction and expected result content.
---
Nitpick comments:
In `@plugins/nemo-experimentalist/examples/smoke-agent/README.md`:
- Around line 15-140: The smoke-agent README currently mixes how-to guidance
with explanation/reference content and timings. Keep the README focused on the
run/check workflow, and move the Design, Layout, Scenarios, and Timings sections
into separate linked pages classified as EXPLANATION or REFERENCE. Update the
existing section anchors and links so the README still points to the relocated
content using the visible README sections as the split points.
🪄 Autofix
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Path: .coderabbit.yaml
Review profile: CHILL
Plan: Enterprise
Run ID: 7c11a54a-71cb-4ce9-923e-2a94bfc083e7
📒 Files selected for processing (177)
plugins/nemo-experimentalist/docs/smoke-agent-weaknesses.mdplugins/nemo-experimentalist/examples/smoke-agent/AGENT-SPEC.mdplugins/nemo-experimentalist/examples/smoke-agent/README.mdplugins/nemo-experimentalist/examples/smoke-agent/agent.pyplugins/nemo-experimentalist/examples/smoke-agent/configs/generalization.yamlplugins/nemo-experimentalist/examples/smoke-agent/configs/repair.yamlplugins/nemo-experimentalist/examples/smoke-agent/dataset/_shared/Dockerfileplugins/nemo-experimentalist/examples/smoke-agent/dataset/_shared/records.jsonplugins/nemo-experimentalist/examples/smoke-agent/dataset/_shared/test.shplugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g1-aggregation/train/lookup-ada/environment/.gitkeepplugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g1-aggregation/train/lookup-ada/instruction.mdplugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g1-aggregation/train/lookup-ada/task.tomlplugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g1-aggregation/train/lookup-ada/tests/expected.txtplugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g1-aggregation/train/lookup-ada/tests/test.shplugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g1-aggregation/train/total-hours-engineers/environment/.gitkeepplugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g1-aggregation/train/total-hours-engineers/instruction.mdplugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g1-aggregation/train/total-hours-engineers/task.tomlplugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g1-aggregation/train/total-hours-engineers/tests/expected.txtplugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g1-aggregation/train/total-hours-engineers/tests/test.shplugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g1-aggregation/train/total-hours-research/environment/.gitkeepplugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g1-aggregation/train/total-hours-research/instruction.mdplugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g1-aggregation/train/total-hours-research/task.tomlplugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g1-aggregation/train/total-hours-research/tests/expected.txtplugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g1-aggregation/train/total-hours-research/tests/test.shplugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g1-aggregation/validation/lookup-grace/environment/.gitkeepplugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g1-aggregation/validation/lookup-grace/instruction.mdplugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g1-aggregation/validation/lookup-grace/task.tomlplugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g1-aggregation/validation/lookup-grace/tests/expected.txtplugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g1-aggregation/validation/lookup-grace/tests/test.shplugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g1-aggregation/validation/total-hours-analysts/environment/.gitkeepplugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g1-aggregation/validation/total-hours-analysts/instruction.mdplugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g1-aggregation/validation/total-hours-analysts/task.tomlplugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g1-aggregation/validation/total-hours-analysts/tests/expected.txtplugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g1-aggregation/validation/total-hours-analysts/tests/test.shplugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g1-aggregation/validation/total-hours-ops/environment/.gitkeepplugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g1-aggregation/validation/total-hours-ops/instruction.mdplugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g1-aggregation/validation/total-hours-ops/task.tomlplugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g1-aggregation/validation/total-hours-ops/tests/expected.txtplugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g1-aggregation/validation/total-hours-ops/tests/test.shplugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g2-name-patterns/train/lookup-ada/environment/.gitkeepplugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g2-name-patterns/train/lookup-ada/instruction.mdplugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g2-name-patterns/train/lookup-ada/task.tomlplugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g2-name-patterns/train/lookup-ada/tests/expected.txtplugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g2-name-patterns/train/lookup-ada/tests/test.shplugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g2-name-patterns/train/lookup-obrien/environment/.gitkeepplugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g2-name-patterns/train/lookup-obrien/instruction.mdplugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g2-name-patterns/train/lookup-obrien/task.tomlplugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g2-name-patterns/train/lookup-obrien/tests/expected.txtplugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g2-name-patterns/train/lookup-obrien/tests/test.shplugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g2-name-patterns/train/lookup-zoe/environment/.gitkeepplugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g2-name-patterns/train/lookup-zoe/instruction.mdplugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g2-name-patterns/train/lookup-zoe/task.tomlplugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g2-name-patterns/train/lookup-zoe/tests/expected.txtplugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g2-name-patterns/train/lookup-zoe/tests/test.shplugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g2-name-patterns/validation/lookup-ann-marie/environment/.gitkeepplugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g2-name-patterns/validation/lookup-ann-marie/instruction.mdplugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g2-name-patterns/validation/lookup-ann-marie/task.tomlplugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g2-name-patterns/validation/lookup-ann-marie/tests/expected.txtplugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g2-name-patterns/validation/lookup-ann-marie/tests/test.shplugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g2-name-patterns/validation/lookup-grace/environment/.gitkeepplugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g2-name-patterns/validation/lookup-grace/instruction.mdplugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g2-name-patterns/validation/lookup-grace/task.tomlplugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g2-name-patterns/validation/lookup-grace/tests/expected.txtplugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g2-name-patterns/validation/lookup-grace/tests/test.shplugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g2-name-patterns/validation/lookup-role-obrien/environment/.gitkeepplugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g2-name-patterns/validation/lookup-role-obrien/instruction.mdplugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g2-name-patterns/validation/lookup-role-obrien/task.tomlplugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g2-name-patterns/validation/lookup-role-obrien/tests/expected.txtplugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g2-name-patterns/validation/lookup-role-obrien/tests/test.shplugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g3-long-inputs/train/plain-dept/environment/.gitkeepplugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g3-long-inputs/train/plain-dept/instruction.mdplugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g3-long-inputs/train/plain-dept/task.tomlplugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g3-long-inputs/train/plain-dept/tests/expected.txtplugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g3-long-inputs/train/plain-dept/tests/test.shplugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g3-long-inputs/train/preamble-dept/environment/.gitkeepplugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g3-long-inputs/train/preamble-dept/instruction.mdplugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g3-long-inputs/train/preamble-dept/task.tomlplugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g3-long-inputs/train/preamble-dept/tests/expected.txtplugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g3-long-inputs/train/preamble-dept/tests/test.shplugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g3-long-inputs/train/preamble-role/environment/.gitkeepplugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g3-long-inputs/train/preamble-role/instruction.mdplugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g3-long-inputs/train/preamble-role/task.tomlplugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g3-long-inputs/train/preamble-role/tests/expected.txtplugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g3-long-inputs/train/preamble-role/tests/test.shplugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g3-long-inputs/validation/preamble-hours/environment/.gitkeepplugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g3-long-inputs/validation/preamble-hours/instruction.mdplugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g3-long-inputs/validation/preamble-hours/task.tomlplugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g3-long-inputs/validation/preamble-hours/tests/expected.txtplugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g3-long-inputs/validation/preamble-hours/tests/test.shplugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g3-long-inputs/validation/preamble-long-dept/environment/.gitkeepplugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g3-long-inputs/validation/preamble-long-dept/instruction.mdplugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g3-long-inputs/validation/preamble-long-dept/task.tomlplugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g3-long-inputs/validation/preamble-long-dept/tests/expected.txtplugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g3-long-inputs/validation/preamble-long-dept/tests/test.shplugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g3-long-inputs/validation/trailing-prose/environment/.gitkeepplugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g3-long-inputs/validation/trailing-prose/instruction.mdplugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g3-long-inputs/validation/trailing-prose/task.tomlplugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g3-long-inputs/validation/trailing-prose/tests/expected.txtplugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g3-long-inputs/validation/trailing-prose/tests/test.shplugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g4-dispatch-order/train/count-ops/environment/.gitkeepplugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g4-dispatch-order/train/count-ops/instruction.mdplugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g4-dispatch-order/train/count-ops/task.tomlplugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g4-dispatch-order/train/count-ops/tests/expected.txtplugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g4-dispatch-order/train/count-ops/tests/test.shplugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g4-dispatch-order/train/count-research/environment/.gitkeepplugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g4-dispatch-order/train/count-research/instruction.mdplugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g4-dispatch-order/train/count-research/task.tomlplugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g4-dispatch-order/train/count-research/tests/expected.txtplugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g4-dispatch-order/train/count-research/tests/test.shplugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g4-dispatch-order/train/lookup-ada/environment/.gitkeepplugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g4-dispatch-order/train/lookup-ada/instruction.mdplugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g4-dispatch-order/train/lookup-ada/task.tomlplugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g4-dispatch-order/train/lookup-ada/tests/expected.txtplugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g4-dispatch-order/train/lookup-ada/tests/test.shplugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g4-dispatch-order/validation/count-engineers-research/environment/.gitkeepplugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g4-dispatch-order/validation/count-engineers-research/instruction.mdplugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g4-dispatch-order/validation/count-engineers-research/task.tomlplugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g4-dispatch-order/validation/count-engineers-research/tests/expected.txtplugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g4-dispatch-order/validation/count-engineers-research/tests/test.shplugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g4-dispatch-order/validation/count-operators-ops/environment/.gitkeepplugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g4-dispatch-order/validation/count-operators-ops/instruction.mdplugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g4-dispatch-order/validation/count-operators-ops/task.tomlplugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g4-dispatch-order/validation/count-operators-ops/tests/expected.txtplugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g4-dispatch-order/validation/count-operators-ops/tests/test.shplugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g4-dispatch-order/validation/lookup-grace/environment/.gitkeepplugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g4-dispatch-order/validation/lookup-grace/instruction.mdplugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g4-dispatch-order/validation/lookup-grace/task.tomlplugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g4-dispatch-order/validation/lookup-grace/tests/expected.txtplugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g4-dispatch-order/validation/lookup-grace/tests/test.shplugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g5-edge-cases/train/empty-role/environment/.gitkeepplugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g5-edge-cases/train/empty-role/instruction.mdplugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g5-edge-cases/train/empty-role/task.tomlplugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g5-edge-cases/train/empty-role/tests/expected.txtplugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g5-edge-cases/train/empty-role/tests/test.shplugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g5-edge-cases/train/lookup-ada/environment/.gitkeepplugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g5-edge-cases/train/lookup-ada/instruction.mdplugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g5-edge-cases/train/lookup-ada/task.tomlplugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g5-edge-cases/train/lookup-ada/tests/expected.txtplugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g5-edge-cases/train/lookup-ada/tests/test.shplugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g5-edge-cases/train/missing-person/environment/.gitkeepplugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g5-edge-cases/train/missing-person/instruction.mdplugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g5-edge-cases/train/missing-person/task.tomlplugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g5-edge-cases/train/missing-person/tests/expected.txtplugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g5-edge-cases/train/missing-person/tests/test.shplugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g5-edge-cases/validation/lookup-grace/environment/.gitkeepplugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g5-edge-cases/validation/lookup-grace/instruction.mdplugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g5-edge-cases/validation/lookup-grace/task.tomlplugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g5-edge-cases/validation/lookup-grace/tests/expected.txtplugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g5-edge-cases/validation/lookup-grace/tests/test.shplugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g5-edge-cases/validation/missing-person-hours/environment/.gitkeepplugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g5-edge-cases/validation/missing-person-hours/instruction.mdplugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g5-edge-cases/validation/missing-person-hours/task.tomlplugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g5-edge-cases/validation/missing-person-hours/tests/expected.txtplugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g5-edge-cases/validation/missing-person-hours/tests/test.shplugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g5-edge-cases/validation/missing-person-role/environment/.gitkeepplugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g5-edge-cases/validation/missing-person-role/instruction.mdplugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g5-edge-cases/validation/missing-person-role/task.tomlplugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g5-edge-cases/validation/missing-person-role/tests/expected.txtplugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g5-edge-cases/validation/missing-person-role/tests/test.shplugins/nemo-experimentalist/examples/smoke-agent/dataset/insights/g1-aggregation.yamlplugins/nemo-experimentalist/examples/smoke-agent/dataset/task-template/README.mdplugins/nemo-experimentalist/examples/smoke-agent/dataset/task-template/environment/.gitkeepplugins/nemo-experimentalist/examples/smoke-agent/dataset/task-template/instruction.mdplugins/nemo-experimentalist/examples/smoke-agent/dataset/task-template/task.tomlplugins/nemo-experimentalist/examples/smoke-agent/dataset/task-template/tests/expected.txtplugins/nemo-experimentalist/examples/smoke-agent/dataset/task-template/tests/test.shplugins/nemo-experimentalist/examples/smoke-agent/harbor_wrapper.pyplugins/nemo-experimentalist/examples/smoke-agent/main.pyplugins/nemo-experimentalist/examples/smoke-agent/optimizer.yamlplugins/nemo-experimentalist/examples/smoke-agent/scripts/build_image.pyplugins/nemo-experimentalist/examples/smoke-agent/scripts/record_traces.pyplugins/nemo-experimentalist/examples/smoke-agent/scripts/sync_verifier.pyplugins/nemo-experimentalist/tests/experimentalist/test_smoke_agent.pyplugins/nemo-experimentalist/tests/experimentalist/test_smoke_agent_assets.pyplugins/nemo-experimentalist/tests/experimentalist/test_smoke_agent_baseline.pyplugins/nemo-experimentalist/tests/experimentalist/test_smoke_agent_gate.pyplugins/nemo-experimentalist/tests/experimentalist/test_smoke_agent_traces.py
Comment on lines
+4
to
+16
| # smoke-agent: the deliberate weaknesses | ||
|
|
||
| `examples/smoke-agent` ships an agent that is **wrong on purpose**. Five known | ||
| weaknesses, each paired with a group of Harbor tasks that surfaces it, so an | ||
| Experimentalist run can be asserted to have *repaired* something rather than | ||
| merely completed. | ||
|
|
||
| **Do not fix them in the agent.** A well-meaning cleanup silently destroys what | ||
| the fixture measures: with the weakness gone, the baseline passes, the Analyzer | ||
| gets no failing trace, and a run that does nothing looks identical to a run that | ||
| works. | ||
|
|
||
| ## Why this file is here and not next to the agent |
Contributor
There was a problem hiding this comment.
📐 Maintainability & Code Quality | 🟡 Minor | ⚡ Quick win
Add the required documentation entry and exit sections.
Identify this page as an EXPLANATION. List prerequisites before the overview. Add a Next Steps section with links to the agent and guard tests.
As per coding guidelines, documentation must list prerequisites and end with a Next Steps section.
Also applies to: 147-151
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@plugins/nemo-experimentalist/docs/smoke-agent-weaknesses.md` around lines 4 -
16, Update the documentation page’s front matter or opening metadata to identify
it as an EXPLANATION, then add a prerequisites section before the existing
overview. Append a Next Steps section at the end linking to the smoke-agent
implementation and its guard tests, preserving the current weakness
documentation.
Source: Coding guidelines
Comment on lines
+31
to
+46
| elif [ -f "$OUTPUT" ]; then | ||
| # Shape check on the first line only: right form, value not considered. `grep -q` | ||
| # is silent on purpose — echoing the line would put answers in the trial log, | ||
| # which the Coder can read. | ||
| if head -n 1 "$OUTPUT" | grep -qE '^[A-Za-z_][A-Za-z0-9_]*='; then | ||
| shape_ok=1.0 | ||
| fi | ||
| # Byte-for-byte over the whole file. Command substitution would strip trailing | ||
| # newlines on both sides, letting an agent append blank lines and still score. | ||
| # CRLF is normalized at end-of-line only: `tr -d '\r'` would delete every CR, | ||
| # so `sum=4<CR>2` would collapse into a passing `sum=42`. | ||
| EXPECTED_NORM="$(mktemp)" | ||
| ACTUAL_NORM="$(mktemp)" | ||
| sed 's/\r$//' "$EXPECTED_FILE" > "$EXPECTED_NORM" | ||
| sed 's/\r$//' "$OUTPUT" > "$ACTUAL_NORM" | ||
| if cmp -s "$EXPECTED_NORM" "$ACTUAL_NORM"; then |
Contributor
There was a problem hiding this comment.
🔒 Security & Privacy | 🟠 Major | 🏗️ Heavy lift
🧩 Analysis chain
🏁 Script executed:
#!/bin/bash
set -eu
for f in \
plugins/nemo-experimentalist/examples/smoke-agent/dataset/_shared/test.sh \
plugins/nemo-experimentalist/examples/smoke-agent/dataset/task-template/tests/test.sh
do
echo "=== $f ==="
sed -n '1,90p' "$f"
done
echo "=== OUTPUT and EXPECTED_FILE references ==="
rg -n -C 3 '(^|[[:space:]])(OUTPUT|EXPECTED_FILE|EXPECTED|/tests|output\.txt|expected\.txt)([=:/"[:space:]]|$)' \
plugins/nemo-experimentalist/examples/smoke-agent/datasetRepository: NVIDIA-NeMo/nemo-platform
Length of output: 50381
🏁 Script executed:
#!/bin/bash
set -eu
echo "=== artifact-path and Harbor references in the relevant subtree ==="
rg -n -m 120 -C 2 '(/app/artifacts|output\.txt|Harbor|agent phase|after the agent)' \
plugins/nemo-experimentalist/examples/smoke-agent \
--glob '!**/test.sh'
echo "=== task-template and shared-file references ==="
rg -n -C 2 '(_shared/test\.sh|task-template|expected\.txt|artifacts)' \
plugins/nemo-experimentalist/examples/smoke-agent/dataset \
--glob '!**/test.sh' \
--glob '!**/expected.txt' | head -n 240Repository: NVIDIA-NeMo/nemo-platform
Length of output: 50384
🏁 Script executed:
#!/bin/bash
set -eu
python3 - <<'PY'
import os
import subprocess
import tempfile
from pathlib import Path
with tempfile.TemporaryDirectory() as d:
root = Path(d)
tests = root / "tests"
artifacts = root / "artifacts"
tests.mkdir()
artifacts.mkdir()
expected = tests / "expected.txt"
output = artifacts / "output.txt"
expected.write_bytes(b"answer=42\n")
output.symlink_to(expected)
commands = [
["test", "-r", str(expected)],
["test", "-f", str(output)],
["head", "-n", "1", str(output)],
]
for command in commands:
result = subprocess.run(command, capture_output=True)
print(" ".join(command[:2]), "status=", result.returncode,
"stdout=", result.stdout.decode().rstrip("\n"))
expected_norm = root / "expected.norm"
actual_norm = root / "actual.norm"
subprocess.run(["sed", "s/\\r$//", str(expected)], stdout=expected_norm.open("wb"), check=True)
subprocess.run(["sed", "s/\\r$//", str(output)], stdout=actual_norm.open("wb"), check=True)
print("cmp_status=", subprocess.run(["cmp", "-s", str(expected_norm), str(actual_norm)]).returncode)
PYRepository: NVIDIA-NeMo/nemo-platform
Length of output: 771
🏁 Script executed:
#!/bin/bash
set -eu
python3 - <<'PY'
import os
import subprocess
import tempfile
from pathlib import Path
with tempfile.TemporaryDirectory() as d:
root = Path(d)
tests = root / "tests"
artifacts = root / "artifacts"
tests.mkdir()
artifacts.mkdir()
expected = tests / "expected.txt"
output = artifacts / "output.txt"
expected.write_bytes(b"answer=42\n")
output.symlink_to(expected)
for command in [
["test", "-r", str(expected)],
["test", "-f", str(output)],
["head", "-n", "1", str(output)],
]:
result = subprocess.run(command, capture_output=True)
print(" ".join(command[:2]), "status=", result.returncode,
"stdout=", result.stdout.decode().rstrip("\n"))
expected_norm = root / "expected.norm"
actual_norm = root / "actual.norm"
with expected_norm.open("wb") as f:
subprocess.run(["sed", "s/\\r$//", str(expected)], stdout=f, check=True)
with actual_norm.open("wb") as f:
subprocess.run(["sed", "s/\\r$//", str(output)], stdout=f, check=True)
print("cmp_status=", subprocess.run(
["cmp", "-s", str(expected_norm), str(actual_norm)]
).returncode)
PYRepository: NVIDIA-NeMo/nemo-platform
Length of output: 771
Other (CWE-59)
Reachability: Internal
Reject symlinked artifacts before scoring.
The agent controls /app/artifacts/output.txt, and Harbor runs the verifier after the agent phase. The verifier follows symlinks in [ -f ], head, sed, and cmp. A link from output.txt to /tests/expected.txt therefore earns reward=1.0. Protect both the shared verifier and the task template by using an isolated finalized artifact or race-safe no-follow file access.
📍 Affects 2 files
plugins/nemo-experimentalist/examples/smoke-agent/dataset/_shared/test.sh#L31-L46(this comment)plugins/nemo-experimentalist/examples/smoke-agent/dataset/task-template/tests/test.sh#L31-L46
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@plugins/nemo-experimentalist/examples/smoke-agent/dataset/_shared/test.sh`
around lines 31 - 46, Reject symlinked output artifacts before scoring in both
plugins/nemo-experimentalist/examples/smoke-agent/dataset/_shared/test.sh lines
31-46 and
plugins/nemo-experimentalist/examples/smoke-agent/dataset/task-template/tests/test.sh
lines 31-46. Update the scoring flow around the OUTPUT file check so validation
uses an isolated finalized artifact or race-safe no-follow access, preventing [
-f ], head, sed, and cmp from following an agent-controlled symlink; preserve
the existing shape and byte-for-byte comparison behavior for regular files.
Contributor
Author
There was a problem hiding this comment.
Confirmed as a real finding, and left open deliberately until it is fixed.
To correct something I said in the summary comment: I suggested the existing symlink guard might already cover this. It does not. harbor_wrapper.py refuses symlinks when uploading from the host into the container; this is the agent creating one inside the container afterwards. Opposite direction, no overlap.
The hole is as described: the agent owns /app/artifacts/output.txt, the verifier reads it after the agent phase, and [ -f ], sed and cmp all follow symlinks — so linking output.txt at the expected fixture would compare the answer key against itself and score 1.0 without solving anything. That defeats the entire premise of this fixture, which is that a reward means something.
Deferred rather than dismissed. The open question is whether the expected fixture is reachable from the agent phase at all; if Harbor only stages tests/ for the verifier, there is nothing to point at and this is theoretical. That needs checking rather than assuming, and the fix if it is reachable is small — refuse a symlinked output before scoring.
Contributor
There was a problem hiding this comment.
🧩 Analysis chain
🏁 Script executed:
#!/bin/bash
set -euo pipefail
echo '=== smoke-agent wrapper outline ==='
ast-grep outline plugins/nemo-experimentalist/examples/smoke-agent/agent/harbor_wrapper.py --items all
echo '=== wrapper implementation ==='
sed -n '1,220p' plugins/nemo-experimentalist/examples/smoke-agent/agent/harbor_wrapper.py
echo '=== /tests and verifier staging references ==='
rg -n -C 4 '(/tests|expected\.txt|verifier|verif|agent phase|run\()' \
plugins/nemo-experimentalist \
--glob '*.py' --glob '*.md' --glob '*.toml' --glob '*.yaml' --glob '*.yml' \
| head -n 400Length of output: 42146
🏁 Script executed:
#!/bin/bash
set -euo pipefail
echo '=== Harbor dependency metadata ==='
rg -n -C 3 '(^harbor\b|harbor[-_]|HarborEvaluator|BaseEnvironment)' \
pyproject.toml uv.lock plugins \
--glob 'pyproject.toml' --glob 'uv.lock' --glob '*.py' \
| head -n 320
echo '=== task lifecycle and fixture staging references across repository ==='
rg -n -C 4 '(/tests|tests/|expected\.txt|verifier_dir|verifier.*run|run.*verifier|setup.*verifier|HarborEvaluator)' \
. \
--glob '!uv.lock' --glob '*.py' --glob '*.md' --glob '*.toml' --glob '*.yaml' --glob '*.yml' \
| head -n 500
echo '=== installed Harbor source, if present ==='
python3 - <<'PY'
import importlib.util
spec = importlib.util.find_spec("harbor")
print(spec.origin if spec else "harbor package not installed")
print(list(spec.submodule_search_locations or []) if spec else "")
PYLength of output: 75147
🏁 Script executed:
#!/bin/bash
set -euo pipefail
echo '=== local Harbor input-path contract ==='
sed -n '90,135p' plugins/nemo-agents/tests/unit/test_improvement_runners.py
sed -n '1,180p' plugins/nemo-agents/src/nemo_agents_plugin/improvement/runners/_harbor_runner.py
echo '=== package metadata required for source lookup ==='
sed -n '2200,2265p' uv.lockLength of output: 16387
🤖 web_search executed:
Harbor Python package 0.20.0 source code verifier /tests agent phase test.sh BaseEnvironment task lifecycle
Length of output: 16244
@schuellc-nvidia Confirmed. harbor_wrapper.py protects only host-to-container uploads. It does not protect agent-created artifacts.
The repository pins Harbor 0.20.0, but does not include its single-step lifecycle implementation. The available evidence is inconclusive on whether /tests/expected.txt exists during this fixture’s agent phase.
Keep this finding open. Verify it with a Harbor 0.20.0 single-step trial that attempts test -r /tests/expected.txt during WrappedAgent.run. If it is reachable, reject [ -L "$OUTPUT" ] before any read or comparison. Apply the same check to the task template verifier.
✏️ Learnings added
Learnt from: schuellc-nvidia
Repo: NVIDIA-NeMo/nemo-platform PR: 1089
File: plugins/nemo-experimentalist/examples/smoke-agent/dataset/_shared/test.sh:31-46
Timestamp: 2026-08-07T13:39:03.265Z
Learning: For the smoke-agent fixture in `plugins/nemo-experimentalist`, `plugins/nemo-experimentalist/examples/smoke-agent/agent/harbor_wrapper.py` rejects symlinks only while uploading agent source files from the host into the task container. It does not protect against a symlink that the agent creates later inside `/app/artifacts/output.txt`. Verifier scripts that read this agent-controlled artifact must independently reject symlinked output before scoring when the verifier fixture is reachable from the agent phase.
Note: Learnings are effective only in the context of similar code segments. To apply general coding guidelines, please configure review instructions. You can manage existing learnings in the app.
🧠 Learnings used
Learnt from: schuellc-nvidia
Repo: NVIDIA-NeMo/nemo-platform PR: 1089
File: plugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g1-aggregation/train/lookup-ada/tests/test.sh:35-36
Timestamp: 2026-08-07T12:31:17.728Z
Learning: In smoke-agent verifier scripts, treat shape_ok as a format metric that only checks whether output has the expected <key>= shape; it must not require a non-empty value. Exact answer correctness is measured separately by reward. Preserve the intentional behavior that role= receives shape_ok=1.0, role=unknown can receive the correct reward, and a fallback sentence fails shape_ok, as enforced by test_shape_metric_discriminates.
You are interacting with an AI system.
Contributor
|
sklinglernv
reviewed
Aug 7, 2026
… the run A proposal round is expensive and everything before it is already paid for, but three separate conditions discarded the whole round rather than the offending improvement. The strictest was a requirement that every improvement in a round carry a distinct optimization_type. Nothing justifies it: a round that wants to edit two different methods is a perfectly good round, and forbidding it also meant a type used once could not be used again later in the run. Two real runs died on it after round 1, having already produced usable candidates. Surplus improvements, a type outside the allowed set, and duplicate optimization text now drop the individual improvement and log why. Only an empty result is fatal, because there is then nothing to build. Callers still pass the available types as a prompt hint, but no longer as a constraint that outlives the round. Adds the first tests for this validation, which had none. Signed-off-by: Christian Schüller <cschueller@nvidia.com>
…turns GroupLeafScorer requires every score to cite span IDs, and the only sanctioned lookups -- get_span_id and get_turn_data -- are indexed by turn. An agent that makes no LLM calls produces a trace with zero turns, so both return None however often they are called. The scorer could not satisfy its own contract, retried to the CodeAct iteration ceiling, and raised GenerationError; because scoring runs inside asyncio.gather with no handler above it, that ended the whole run -- discarding candidates that had already been evaluated successfully. Nothing was wrong with the evidence. A turn-less trace still carries a full call graph: which methods ran, in what order, and their status, which is enough to tell a broken candidate from a fixed one. The scorer could see it; it just had no permitted way to cite it, since the ids visible in the overview text are abbreviated and explicitly disallowed. The contract now names the observable condition (`Turns: 0`), says to leave span_ids empty, and points at the call graph as the grounding to use instead. Guessed or abbreviated ids are still refused, and traces that do have turns are unaffected. Verified on a run against a deterministic agent: the scorer completed and separated two candidates that were tied on reward, citing method names and statuses. Not specific to one fixture -- any deterministic, rule-based, or purely tool-driven agent produces turn-less traces and hit the same dead end. Signed-off-by: Christian Schüller <cschueller@nvidia.com>
The existing examples take upwards of two hours per run and produce stochastic outcomes, so they can confirm the loop completed but never that it improved anything. This is a fixture where the expected result is known in advance, which makes a run checkable rather than merely plausible. The agent makes no model calls -- handlers are regular expressions over a small records file -- so the only stochastic component in a run is the Experimentalist itself, and its CompletionClient points at an unroutable address so an accidental model call fails loudly rather than quietly making the agent nondeterministic. One prebuilt image serves every task, tagged by a content hash of the Dockerfile and records, so a forgotten rebuild fails a test instead of silently running against stale data. Five task groups, each answering a different question about the loop: a repair that a general fix reaches, a generalization trap where the tempting fix passes train and fails validation, a pattern too narrow, a clipped instruction, and a fix that needs several coordinated edits. Every group carries a control that passes at baseline, so a destructive fix costs reward instead of going unnoticed. Three scenarios: repair and generalization run a single round and assert opposite outcomes, and a full multi-round scenario runs several groups at once against a generated combined dataset to exercise survivors, ranking and the convergence check. Two groups are held out of the combined set with the reasons recorded in build_all_group.py. What the fixture measures is documented outside the copied surface on purpose. `agent_source` points at `agent/`, which holds the agent and nothing else, so the README, configs, scripts and dataset are never copied into a candidate workspace where the Coder would read them -- the boundary is structural rather than a list of exclusions, and a test fails if it is widened. Verified end to end: the repair scenario reaches validation 1.000 from a 0.333 baseline, the generalization scenario correctly retains the baseline, and the full scenario improves across rounds and converges on its own. Signed-off-by: Christian Schüller <cschueller@nvidia.com>
schuellc-nvidia
force-pushed
the
experimentalist-smoke-agent-proposal/cschueller
branch
from
August 7, 2026 12:18
3ba1d3e to
01ddc0e
Compare
Contributor
|
Note GitHub couldn't provide a complete incremental comparison for this pull request, so CodeRabbit is performing a full review instead. This review may take a little longer. |
schuellc-nvidia
changed the base branch from
main
to
experimentalist-fixes/cschueller
August 7, 2026 12:20
8 tasks
Contributor
Author
Review statusNine threads answered and resolved individually. The remaining twelve are Deferred, needing a decisionSPDX headers on task fixtures (8 threads). These need a call rather than a
Either the fixture format grows a metadata section the verifier and loader strip, Symlink handling, CWE-59 (2 threads).
Entry/exit sections in the weaknesses doc (1 thread). Straightforward; not Also worth notingThis branch was force-pushed after a rebase onto current The two Experimentalist fixes this fixture depends on now live in #1163, and this |
Contributor
There was a problem hiding this comment.
Actionable comments posted: 13
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@plugins/nemo-experimentalist/examples/smoke-agent/.gitignore`:
- Around line 1-7: Add the required NVIDIA SPDX copyright header and Apache-2.0
license identifier at the top of the .gitignore file, before the existing
generated-dataset comments, while preserving the ignore rule and explanatory
text.
In `@plugins/nemo-experimentalist/examples/smoke-agent/agent/agent.py`:
- Line 60: Define a concrete TypedDict for records with name, dept, role, and
hours fields, then update the _records annotation in the agent initialization to
use list of that TypedDict instead of list[dict].
In `@plugins/nemo-experimentalist/examples/smoke-agent/agent/harbor_wrapper.py`:
- Around line 87-101: The upload flow around _reject_symlinks and the
selected-entry loop must apply EXCLUDE and EXCLUDE_GLOB recursively, not only to
AGENT_DIR’s top-level children. Build an allowlisted staging tree or otherwise
ensure every descendant is filtered before environment.upload_dir or
environment.upload_file, preventing nested credentials and excluded directories
such as .env and .venv from reaching the container.
- Around line 112-115: Update the command constructed in the environment.exec
call to invoke main.py through uv using “uv run python main.py”, while
preserving the existing quoted instruction and session_id arguments. Do not add
--no-project or --frozen.
In
`@plugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g2-name-patterns/validation/lookup-ann-marie/instruction.md`:
- Around line 1-9: Define a non-semantic SPDX metadata mechanism for all five
instruction.md fixtures without adding literal headers to prompt content:
plugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g2-name-patterns/validation/lookup-ann-marie/instruction.md
(lines 1-9), lookup-grace/instruction.md (lines 1-9),
lookup-role-obrien/instruction.md (lines 1-9),
plugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g3-long-inputs/train/plain-dept/instruction.md
(lines 1-9), and validation/trailing-prose/instruction.md (lines 1-9). Store the
required SPDX lines outside the prompts or implement a defined metadata-block
stripping path in the instruction loader, and add tests confirming the loaded
prompt content and baseline behavior remain unchanged.
In
`@plugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g4-dispatch-order/train/count-research/instruction.md`:
- Line 1: Add Markdown SPDX copyright and Apache-2.0 license comments before the
existing prompt in each listed instruction.md file:
plugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g4-dispatch-order/train/count-research/instruction.md
(lines 1-1),
plugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g4-dispatch-order/train/lookup-ada/instruction.md
(lines 1-1),
plugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g4-dispatch-order/validation/count-engineers-research/instruction.md
(lines 1-1),
plugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g4-dispatch-order/validation/count-operators-ops/instruction.md
(lines 1-1),
plugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g4-dispatch-order/validation/lookup-grace/instruction.md
(lines 1-1), and
plugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g5-edge-cases/train/empty-role/instruction.md
(lines 1-1). Preserve each instruction’s prompt content exactly and validate
task behavior after adding the metadata.
In
`@plugins/nemo-experimentalist/examples/smoke-agent/dataset/task-template/instruction.md`:
- Around line 1-9: Add non-rendered NVIDIA copyright and Apache-2.0 SPDX
identifier comments to each affected instruction.md fixture, preserving every
existing task prompt and required output format:
plugins/nemo-experimentalist/examples/smoke-agent/dataset/task-template/instruction.md
lines 1-9;
plugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g1-aggregation/validation/total-hours-analysts/instruction.md
lines 1-9;
plugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g1-aggregation/validation/total-hours-ops/instruction.md
lines 1-9;
plugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g5-edge-cases/train/missing-person/instruction.md
lines 1-9;
plugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g5-edge-cases/validation/lookup-grace/instruction.md
lines 1-9;
plugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g5-edge-cases/validation/missing-person-hours/instruction.md
lines 1-9; and
plugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g5-edge-cases/validation/missing-person-role/instruction.md
lines 1-9.
In `@plugins/nemo-experimentalist/examples/smoke-agent/README.md`:
- Around line 199-202: Update the gate description near the existing “Five
assertions” text to state that it contains eight assertions, including rejection
of candidates that fail generalization on held-out instances. Ensure the listed
assertion categories accurately cover all eight checks.
In
`@plugins/nemo-experimentalist/examples/smoke-agent/scripts/build_all_group.py`:
- Around line 64-68: Update assemble so removal of the generated target
directory does not suppress deletion errors: replace the ignore_errors behavior
in the shutil.rmtree call with error propagation. Keep the existing target path
and rebuild flow unchanged so assembly stops instead of continuing with stale
contents when deletion is incomplete.
In
`@plugins/nemo-experimentalist/src/nemo_experimentalist_plugin/experimentalist/components/proposer.py`:
- Around line 163-170: Update the proposer prompt construction in
_run_with_context so available_types are presented as a preference rather than a
requirement, allowing types from tried_types to be proposed when appropriate.
Preserve the all_types allowance used by _filter_improvements, and add a prompt
contract test verifying reused types remain permissible while untried types are
favored.
In
`@plugins/nemo-experimentalist/tests/experimentalist/test_smoke_agent_assets.py`:
- Around line 39-41: Update `_task_tomls()` to assert that `dataset/groups`
exists and that its recursive `task.toml` result is non-empty, failing when task
fixtures are missing. Remove early returns in the image, verifier, and
environment tests that skip execution for an empty task list, while preserving
`_all` as optional in its separate tests.
In `@plugins/nemo-experimentalist/tests/experimentalist/test_smoke_agent_gate.py`:
- Around line 108-127: Update _agent_class so candidate agent.py files are never
imported or executed in the pytest process. Remove the direct importlib
execution path and replay each candidate through the existing isolated Harbor or
container environment, returning its ReportAgent-compatible result via that
boundary; do not use a host subprocess as a substitute for isolation.
In
`@plugins/nemo-experimentalist/tests/experimentalist/test_trace_scorer_contract.py`:
- Around line 43-53: Strengthen test_contract_offers_an_alternative_grounding by
asserting the contract includes “name the methods that ran”, “the order they ran
in”, “their status”, and “do not retry” in addition to the existing call-graph
requirement. Keep the zero-turn grounding and forbidden-ID assertions unchanged.
🪄 Autofix
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Path: .coderabbit.yaml
Review profile: CHILL
Plan: Enterprise
Run ID: 12efb76a-62cb-4d47-8575-f0aa26940538
📒 Files selected for processing (184)
plugins/nemo-experimentalist/docs/smoke-agent-weaknesses.mdplugins/nemo-experimentalist/examples/smoke-agent/.gitignoreplugins/nemo-experimentalist/examples/smoke-agent/AGENT-SPEC.mdplugins/nemo-experimentalist/examples/smoke-agent/README.mdplugins/nemo-experimentalist/examples/smoke-agent/agent/agent.pyplugins/nemo-experimentalist/examples/smoke-agent/agent/harbor_wrapper.pyplugins/nemo-experimentalist/examples/smoke-agent/agent/main.pyplugins/nemo-experimentalist/examples/smoke-agent/configs/full.yamlplugins/nemo-experimentalist/examples/smoke-agent/configs/generalization.yamlplugins/nemo-experimentalist/examples/smoke-agent/configs/repair.yamlplugins/nemo-experimentalist/examples/smoke-agent/dataset/_shared/Dockerfileplugins/nemo-experimentalist/examples/smoke-agent/dataset/_shared/records.jsonplugins/nemo-experimentalist/examples/smoke-agent/dataset/_shared/test.shplugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g1-aggregation/train/lookup-ada/environment/.gitkeepplugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g1-aggregation/train/lookup-ada/instruction.mdplugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g1-aggregation/train/lookup-ada/task.tomlplugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g1-aggregation/train/lookup-ada/tests/expected.txtplugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g1-aggregation/train/lookup-ada/tests/test.shplugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g1-aggregation/train/total-hours-engineers/environment/.gitkeepplugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g1-aggregation/train/total-hours-engineers/instruction.mdplugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g1-aggregation/train/total-hours-engineers/task.tomlplugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g1-aggregation/train/total-hours-engineers/tests/expected.txtplugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g1-aggregation/train/total-hours-engineers/tests/test.shplugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g1-aggregation/train/total-hours-research/environment/.gitkeepplugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g1-aggregation/train/total-hours-research/instruction.mdplugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g1-aggregation/train/total-hours-research/task.tomlplugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g1-aggregation/train/total-hours-research/tests/expected.txtplugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g1-aggregation/train/total-hours-research/tests/test.shplugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g1-aggregation/validation/lookup-grace/environment/.gitkeepplugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g1-aggregation/validation/lookup-grace/instruction.mdplugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g1-aggregation/validation/lookup-grace/task.tomlplugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g1-aggregation/validation/lookup-grace/tests/expected.txtplugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g1-aggregation/validation/lookup-grace/tests/test.shplugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g1-aggregation/validation/total-hours-analysts/environment/.gitkeepplugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g1-aggregation/validation/total-hours-analysts/instruction.mdplugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g1-aggregation/validation/total-hours-analysts/task.tomlplugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g1-aggregation/validation/total-hours-analysts/tests/expected.txtplugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g1-aggregation/validation/total-hours-analysts/tests/test.shplugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g1-aggregation/validation/total-hours-ops/environment/.gitkeepplugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g1-aggregation/validation/total-hours-ops/instruction.mdplugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g1-aggregation/validation/total-hours-ops/task.tomlplugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g1-aggregation/validation/total-hours-ops/tests/expected.txtplugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g1-aggregation/validation/total-hours-ops/tests/test.shplugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g2-name-patterns/train/lookup-ada/environment/.gitkeepplugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g2-name-patterns/train/lookup-ada/instruction.mdplugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g2-name-patterns/train/lookup-ada/task.tomlplugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g2-name-patterns/train/lookup-ada/tests/expected.txtplugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g2-name-patterns/train/lookup-ada/tests/test.shplugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g2-name-patterns/train/lookup-obrien/environment/.gitkeepplugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g2-name-patterns/train/lookup-obrien/instruction.mdplugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g2-name-patterns/train/lookup-obrien/task.tomlplugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g2-name-patterns/train/lookup-obrien/tests/expected.txtplugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g2-name-patterns/train/lookup-obrien/tests/test.shplugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g2-name-patterns/train/lookup-zoe/environment/.gitkeepplugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g2-name-patterns/train/lookup-zoe/instruction.mdplugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g2-name-patterns/train/lookup-zoe/task.tomlplugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g2-name-patterns/train/lookup-zoe/tests/expected.txtplugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g2-name-patterns/train/lookup-zoe/tests/test.shplugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g2-name-patterns/validation/lookup-ann-marie/environment/.gitkeepplugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g2-name-patterns/validation/lookup-ann-marie/instruction.mdplugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g2-name-patterns/validation/lookup-ann-marie/task.tomlplugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g2-name-patterns/validation/lookup-ann-marie/tests/expected.txtplugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g2-name-patterns/validation/lookup-ann-marie/tests/test.shplugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g2-name-patterns/validation/lookup-grace/environment/.gitkeepplugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g2-name-patterns/validation/lookup-grace/instruction.mdplugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g2-name-patterns/validation/lookup-grace/task.tomlplugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g2-name-patterns/validation/lookup-grace/tests/expected.txtplugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g2-name-patterns/validation/lookup-grace/tests/test.shplugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g2-name-patterns/validation/lookup-role-obrien/environment/.gitkeepplugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g2-name-patterns/validation/lookup-role-obrien/instruction.mdplugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g2-name-patterns/validation/lookup-role-obrien/task.tomlplugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g2-name-patterns/validation/lookup-role-obrien/tests/expected.txtplugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g2-name-patterns/validation/lookup-role-obrien/tests/test.shplugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g3-long-inputs/train/plain-dept/environment/.gitkeepplugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g3-long-inputs/train/plain-dept/instruction.mdplugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g3-long-inputs/train/plain-dept/task.tomlplugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g3-long-inputs/train/plain-dept/tests/expected.txtplugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g3-long-inputs/train/plain-dept/tests/test.shplugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g3-long-inputs/train/preamble-dept/environment/.gitkeepplugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g3-long-inputs/train/preamble-dept/instruction.mdplugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g3-long-inputs/train/preamble-dept/task.tomlplugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g3-long-inputs/train/preamble-dept/tests/expected.txtplugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g3-long-inputs/train/preamble-dept/tests/test.shplugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g3-long-inputs/train/preamble-role/environment/.gitkeepplugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g3-long-inputs/train/preamble-role/instruction.mdplugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g3-long-inputs/train/preamble-role/task.tomlplugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g3-long-inputs/train/preamble-role/tests/expected.txtplugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g3-long-inputs/train/preamble-role/tests/test.shplugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g3-long-inputs/validation/preamble-hours/environment/.gitkeepplugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g3-long-inputs/validation/preamble-hours/instruction.mdplugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g3-long-inputs/validation/preamble-hours/task.tomlplugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g3-long-inputs/validation/preamble-hours/tests/expected.txtplugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g3-long-inputs/validation/preamble-hours/tests/test.shplugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g3-long-inputs/validation/preamble-long-dept/environment/.gitkeepplugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g3-long-inputs/validation/preamble-long-dept/instruction.mdplugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g3-long-inputs/validation/preamble-long-dept/task.tomlplugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g3-long-inputs/validation/preamble-long-dept/tests/expected.txtplugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g3-long-inputs/validation/preamble-long-dept/tests/test.shplugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g3-long-inputs/validation/trailing-prose/environment/.gitkeepplugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g3-long-inputs/validation/trailing-prose/instruction.mdplugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g3-long-inputs/validation/trailing-prose/task.tomlplugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g3-long-inputs/validation/trailing-prose/tests/expected.txtplugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g3-long-inputs/validation/trailing-prose/tests/test.shplugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g4-dispatch-order/train/count-ops/environment/.gitkeepplugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g4-dispatch-order/train/count-ops/instruction.mdplugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g4-dispatch-order/train/count-ops/task.tomlplugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g4-dispatch-order/train/count-ops/tests/expected.txtplugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g4-dispatch-order/train/count-ops/tests/test.shplugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g4-dispatch-order/train/count-research/environment/.gitkeepplugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g4-dispatch-order/train/count-research/instruction.mdplugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g4-dispatch-order/train/count-research/task.tomlplugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g4-dispatch-order/train/count-research/tests/expected.txtplugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g4-dispatch-order/train/count-research/tests/test.shplugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g4-dispatch-order/train/lookup-ada/environment/.gitkeepplugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g4-dispatch-order/train/lookup-ada/instruction.mdplugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g4-dispatch-order/train/lookup-ada/task.tomlplugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g4-dispatch-order/train/lookup-ada/tests/expected.txtplugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g4-dispatch-order/train/lookup-ada/tests/test.shplugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g4-dispatch-order/validation/count-engineers-research/environment/.gitkeepplugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g4-dispatch-order/validation/count-engineers-research/instruction.mdplugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g4-dispatch-order/validation/count-engineers-research/task.tomlplugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g4-dispatch-order/validation/count-engineers-research/tests/expected.txtplugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g4-dispatch-order/validation/count-engineers-research/tests/test.shplugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g4-dispatch-order/validation/count-operators-ops/environment/.gitkeepplugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g4-dispatch-order/validation/count-operators-ops/instruction.mdplugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g4-dispatch-order/validation/count-operators-ops/task.tomlplugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g4-dispatch-order/validation/count-operators-ops/tests/expected.txtplugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g4-dispatch-order/validation/count-operators-ops/tests/test.shplugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g4-dispatch-order/validation/lookup-grace/environment/.gitkeepplugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g4-dispatch-order/validation/lookup-grace/instruction.mdplugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g4-dispatch-order/validation/lookup-grace/task.tomlplugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g4-dispatch-order/validation/lookup-grace/tests/expected.txtplugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g4-dispatch-order/validation/lookup-grace/tests/test.shplugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g5-edge-cases/train/empty-role/environment/.gitkeepplugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g5-edge-cases/train/empty-role/instruction.mdplugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g5-edge-cases/train/empty-role/task.tomlplugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g5-edge-cases/train/empty-role/tests/expected.txtplugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g5-edge-cases/train/empty-role/tests/test.shplugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g5-edge-cases/train/lookup-ada/environment/.gitkeepplugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g5-edge-cases/train/lookup-ada/instruction.mdplugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g5-edge-cases/train/lookup-ada/task.tomlplugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g5-edge-cases/train/lookup-ada/tests/expected.txtplugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g5-edge-cases/train/lookup-ada/tests/test.shplugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g5-edge-cases/train/missing-person/environment/.gitkeepplugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g5-edge-cases/train/missing-person/instruction.mdplugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g5-edge-cases/train/missing-person/task.tomlplugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g5-edge-cases/train/missing-person/tests/expected.txtplugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g5-edge-cases/train/missing-person/tests/test.shplugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g5-edge-cases/validation/lookup-grace/environment/.gitkeepplugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g5-edge-cases/validation/lookup-grace/instruction.mdplugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g5-edge-cases/validation/lookup-grace/task.tomlplugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g5-edge-cases/validation/lookup-grace/tests/expected.txtplugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g5-edge-cases/validation/lookup-grace/tests/test.shplugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g5-edge-cases/validation/missing-person-hours/environment/.gitkeepplugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g5-edge-cases/validation/missing-person-hours/instruction.mdplugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g5-edge-cases/validation/missing-person-hours/task.tomlplugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g5-edge-cases/validation/missing-person-hours/tests/expected.txtplugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g5-edge-cases/validation/missing-person-hours/tests/test.shplugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g5-edge-cases/validation/missing-person-role/environment/.gitkeepplugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g5-edge-cases/validation/missing-person-role/instruction.mdplugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g5-edge-cases/validation/missing-person-role/task.tomlplugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g5-edge-cases/validation/missing-person-role/tests/expected.txtplugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g5-edge-cases/validation/missing-person-role/tests/test.shplugins/nemo-experimentalist/examples/smoke-agent/dataset/insights/g1-aggregation.yamlplugins/nemo-experimentalist/examples/smoke-agent/dataset/task-template/README.mdplugins/nemo-experimentalist/examples/smoke-agent/dataset/task-template/environment/.gitkeepplugins/nemo-experimentalist/examples/smoke-agent/dataset/task-template/instruction.mdplugins/nemo-experimentalist/examples/smoke-agent/dataset/task-template/task.tomlplugins/nemo-experimentalist/examples/smoke-agent/dataset/task-template/tests/expected.txtplugins/nemo-experimentalist/examples/smoke-agent/dataset/task-template/tests/test.shplugins/nemo-experimentalist/examples/smoke-agent/optimizer.yamlplugins/nemo-experimentalist/examples/smoke-agent/scripts/build_all_group.pyplugins/nemo-experimentalist/examples/smoke-agent/scripts/build_image.pyplugins/nemo-experimentalist/examples/smoke-agent/scripts/record_traces.pyplugins/nemo-experimentalist/examples/smoke-agent/scripts/sync_verifier.pyplugins/nemo-experimentalist/src/nemo_experimentalist_plugin/experimentalist/components/proposer.pyplugins/nemo-experimentalist/src/nemo_experimentalist_plugin/experimentalist/components/trace_scorer.pyplugins/nemo-experimentalist/tests/experimentalist/test_proposer_validation.pyplugins/nemo-experimentalist/tests/experimentalist/test_smoke_agent.pyplugins/nemo-experimentalist/tests/experimentalist/test_smoke_agent_assets.pyplugins/nemo-experimentalist/tests/experimentalist/test_smoke_agent_baseline.pyplugins/nemo-experimentalist/tests/experimentalist/test_smoke_agent_gate.pyplugins/nemo-experimentalist/tests/experimentalist/test_smoke_agent_traces.pyplugins/nemo-experimentalist/tests/experimentalist/test_trace_scorer_contract.py
🚧 Files skipped from review as they are similar to previous changes (99)
- plugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g3-long-inputs/validation/trailing-prose/tests/expected.txt
- plugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g3-long-inputs/train/preamble-dept/tests/expected.txt
- plugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g5-edge-cases/validation/missing-person-role/task.toml
- plugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g1-aggregation/train/total-hours-research/task.toml
- plugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g5-edge-cases/validation/missing-person-role/tests/expected.txt
- plugins/nemo-experimentalist/examples/smoke-agent/dataset/task-template/task.toml
- plugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g1-aggregation/validation/lookup-grace/task.toml
- plugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g3-long-inputs/validation/preamble-long-dept/tests/expected.txt
- plugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g4-dispatch-order/train/lookup-ada/tests/expected.txt
- plugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g1-aggregation/train/total-hours-engineers/tests/expected.txt
- plugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g1-aggregation/train/total-hours-research/tests/expected.txt
- plugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g2-name-patterns/train/lookup-ada/tests/expected.txt
- plugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g2-name-patterns/validation/lookup-grace/tests/test.sh
- plugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g4-dispatch-order/validation/lookup-grace/task.toml
- plugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g4-dispatch-order/train/count-research/tests/expected.txt
- plugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g3-long-inputs/train/plain-dept/tests/test.sh
- plugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g2-name-patterns/train/lookup-ada/task.toml
- plugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g2-name-patterns/train/lookup-zoe/task.toml
- plugins/nemo-experimentalist/examples/smoke-agent/dataset/task-template/tests/test.sh
- plugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g1-aggregation/train/lookup-ada/task.toml
- plugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g2-name-patterns/validation/lookup-grace/tests/expected.txt
- plugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g5-edge-cases/train/missing-person/tests/expected.txt
- plugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g5-edge-cases/validation/missing-person-hours/task.toml
- plugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g4-dispatch-order/validation/count-engineers-research/tests/expected.txt
- plugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g5-edge-cases/train/missing-person/task.toml
- plugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g2-name-patterns/validation/lookup-ann-marie/tests/expected.txt
- plugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g4-dispatch-order/train/count-ops/tests/test.sh
- plugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g4-dispatch-order/train/count-research/task.toml
- plugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g5-edge-cases/train/empty-role/tests/expected.txt
- plugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g3-long-inputs/train/preamble-role/task.toml
- plugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g4-dispatch-order/validation/count-operators-ops/tests/test.sh
- plugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g1-aggregation/validation/lookup-grace/tests/expected.txt
- plugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g2-name-patterns/validation/lookup-role-obrien/tests/test.sh
- plugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g5-edge-cases/validation/lookup-grace/task.toml
- plugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g2-name-patterns/train/lookup-obrien/tests/expected.txt
- plugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g5-edge-cases/train/lookup-ada/task.toml
- plugins/nemo-experimentalist/examples/smoke-agent/dataset/_shared/test.sh
- plugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g4-dispatch-order/validation/lookup-grace/tests/expected.txt
- plugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g4-dispatch-order/train/count-ops/task.toml
- plugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g1-aggregation/validation/total-hours-analysts/tests/test.sh
- plugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g1-aggregation/train/total-hours-engineers/task.toml
- plugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g1-aggregation/validation/total-hours-ops/tests/expected.txt
- plugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g4-dispatch-order/validation/lookup-grace/tests/test.sh
- plugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g1-aggregation/validation/lookup-grace/tests/test.sh
- plugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g5-edge-cases/validation/lookup-grace/tests/test.sh
- plugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g2-name-patterns/validation/lookup-ann-marie/tests/test.sh
- plugins/nemo-experimentalist/examples/smoke-agent/dataset/task-template/README.md
- plugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g5-edge-cases/validation/missing-person-role/tests/test.sh
- plugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g2-name-patterns/train/lookup-zoe/tests/test.sh
- plugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g3-long-inputs/validation/preamble-hours/tests/test.sh
- plugins/nemo-experimentalist/examples/smoke-agent/optimizer.yaml
- plugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g4-dispatch-order/train/lookup-ada/task.toml
- plugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g3-long-inputs/train/preamble-dept/task.toml
- plugins/nemo-experimentalist/examples/smoke-agent/dataset/task-template/tests/expected.txt
- plugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g5-edge-cases/train/lookup-ada/tests/expected.txt
- plugins/nemo-experimentalist/examples/smoke-agent/dataset/_shared/records.json
- plugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g3-long-inputs/train/preamble-dept/tests/test.sh
- plugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g2-name-patterns/validation/lookup-role-obrien/tests/expected.txt
- plugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g2-name-patterns/train/lookup-zoe/tests/expected.txt
- plugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g5-edge-cases/validation/missing-person-hours/tests/expected.txt
- plugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g4-dispatch-order/validation/count-engineers-research/task.toml
- plugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g1-aggregation/train/total-hours-engineers/tests/test.sh
- plugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g2-name-patterns/validation/lookup-ann-marie/task.toml
- plugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g4-dispatch-order/validation/count-operators-ops/task.toml
- plugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g3-long-inputs/validation/preamble-long-dept/task.toml
- plugins/nemo-experimentalist/tests/experimentalist/test_smoke_agent.py
- plugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g1-aggregation/train/lookup-ada/tests/expected.txt
- plugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g2-name-patterns/validation/lookup-grace/task.toml
- plugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g2-name-patterns/validation/lookup-role-obrien/task.toml
- plugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g1-aggregation/validation/total-hours-ops/task.toml
- plugins/nemo-experimentalist/examples/smoke-agent/scripts/sync_verifier.py
- plugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g1-aggregation/validation/total-hours-ops/tests/test.sh
- plugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g3-long-inputs/train/plain-dept/tests/expected.txt
- plugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g3-long-inputs/train/preamble-role/tests/test.sh
- plugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g2-name-patterns/train/lookup-ada/tests/test.sh
- plugins/nemo-experimentalist/examples/smoke-agent/configs/generalization.yaml
- plugins/nemo-experimentalist/docs/smoke-agent-weaknesses.md
- plugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g3-long-inputs/validation/trailing-prose/task.toml
- plugins/nemo-experimentalist/examples/smoke-agent/scripts/record_traces.py
- plugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g4-dispatch-order/train/count-ops/tests/expected.txt
- plugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g3-long-inputs/validation/preamble-hours/task.toml
- plugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g5-edge-cases/train/empty-role/tests/test.sh
- plugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g4-dispatch-order/validation/count-engineers-research/tests/test.sh
- plugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g5-edge-cases/train/empty-role/task.toml
- plugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g3-long-inputs/validation/preamble-long-dept/tests/test.sh
- plugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g4-dispatch-order/validation/count-operators-ops/tests/expected.txt
- plugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g3-long-inputs/train/plain-dept/task.toml
- plugins/nemo-experimentalist/examples/smoke-agent/dataset/insights/g1-aggregation.yaml
- plugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g3-long-inputs/train/preamble-role/tests/expected.txt
- plugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g2-name-patterns/train/lookup-obrien/tests/test.sh
- plugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g2-name-patterns/train/lookup-obrien/task.toml
- plugins/nemo-experimentalist/tests/experimentalist/test_smoke_agent_baseline.py
- plugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g3-long-inputs/validation/trailing-prose/tests/test.sh
- plugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g5-edge-cases/validation/lookup-grace/tests/expected.txt
- plugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g1-aggregation/validation/total-hours-analysts/tests/expected.txt
- plugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g3-long-inputs/validation/preamble-hours/tests/expected.txt
- plugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g4-dispatch-order/train/lookup-ada/tests/test.sh
- plugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g4-dispatch-order/train/count-research/tests/test.sh
- plugins/nemo-experimentalist/examples/smoke-agent/configs/repair.yaml
Contributor
There was a problem hiding this comment.
Caution
Inline review comments failed to post. This is likely due to GitHub's internal server error or limits when posting large numbers of comments. If you are seeing this consistently it is likely a permissions issue. Please check "Moderation" -> "Code review limits" under your organization settings.
🛑 Comments failed to post (13)
plugins/nemo-experimentalist/examples/smoke-agent/.gitignore (1)
1-7: 📐 Maintainability & Code Quality | 🟡 Minor | ⚡ Quick win
Add the required SPDX header.
This file lacks the NVIDIA SPDX copyright header and Apache-2.0 license identifier.
As per coding guidelines: “Every file must include the required NVIDIA SPDX copyright header and Apache-2.0 license identifier.”
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@plugins/nemo-experimentalist/examples/smoke-agent/.gitignore` around lines 1 - 7, Add the required NVIDIA SPDX copyright header and Apache-2.0 license identifier at the top of the .gitignore file, before the existing generated-dataset comments, while preserving the ignore rule and explanatory text.Source: Coding guidelines
plugins/nemo-experimentalist/examples/smoke-agent/agent/agent.py (1)
60-60: 📐 Maintainability & Code Quality | 🟡 Minor | ⚡ Quick win
🧩 Analysis chain
🏁 Script executed:
#!/bin/bash set -eu file=$(fd -t f 'agent\.py$' plugins/nemo-experimentalist/examples/smoke-agent) printf '%s\n' "$file" sed -n '1,130p' "$file" printf '\n--- record access and related types ---\n' rg -n --glob '*.py' '(_records|RECORDS_PATH|TypedDict|Record)' plugins/nemo-experimentalist/examples/smoke-agent plugins/nemo-experimentalistRepository: NVIDIA-NeMo/nemo-platform
Length of output: 22120
🏁 Script executed:
#!/bin/bash set -eu printf '%s\n' '--- smoke-agent files ---' git ls-files plugins/nemo-experimentalist/examples/smoke-agent printf '%s\n' '--- record fixtures ---' fd -t f -i 'records.*' plugins/nemo-experimentalist printf '%s\n' '--- baseline fixture and configuration ---' sed -n '1,95p' plugins/nemo-experimentalist/tests/experimentalist/test_smoke_agent_baseline.py rg -n --glob 'pyproject.toml' --glob '*.toml' 'ruff|ty|mypy|disallow_any|ann' plugins/nemo-experimentalist . printf '%s\n' '--- JSON record shape probe ---' python3 - <<'PY' import json from pathlib import Path for path in Path("plugins/nemo-experimentalist").rglob("*.json"): if "records" not in path.name.lower(): continue try: value = json.loads(path.read_text(encoding="utf-8")) except (OSError, json.JSONDecodeError): continue print(path) print("root_type:", type(value).__name__, "length:", len(value) if hasattr(value, "__len__") else None) if isinstance(value, list): for index, record in enumerate(value): print(index, type(record).__name__, sorted(record) if isinstance(record, dict) else None, {key: type(item).__name__ for key, item in record.items()} if isinstance(record, dict) else None) PYRepository: NVIDIA-NeMo/nemo-platform
Length of output: 50381
🏁 Script executed:
#!/bin/bash set -eu record_file=plugins/nemo-experimentalist/examples/smoke-agent/dataset/_shared/records.json printf '%s\n' '--- records.json ---' cat "$record_file" printf '%s\n' '--- targeted type-check configuration ---' sed -n '560,690p' pyproject.toml printf '%s\n' '--- root Ruff rules ---' sed -n '1,130p' ruff.toml printf '%s\n' '--- record-shape verifier ---' python3 - <<'PY' import json from pathlib import Path path = Path("plugins/nemo-experimentalist/examples/smoke-agent/dataset/_shared/records.json") records = json.loads(path.read_text(encoding="utf-8")) assert isinstance(records, list) expected = {"name": str, "dept": str, "role": str, "hours": int} print("records:", len(records)) for index, record in enumerate(records): assert isinstance(record, dict), (index, type(record).__name__) print(index, {key: type(value).__name__ for key, value in record.items()}) assert set(record) == set(expected), (index, set(record)) assert all(isinstance(record[key], expected[key]) for key in expected), index print("shape: list[TypedDict-like records] with keys name, dept, role, hours") PYRepository: NVIDIA-NeMo/nemo-platform
Length of output: 8357
Use a concrete record type for
_records.Define a
TypedDictwithname,dept,role, andhours, then annotate_recordswith it instead oflist[dict].🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@plugins/nemo-experimentalist/examples/smoke-agent/agent/agent.py` at line 60, Define a concrete TypedDict for records with name, dept, role, and hours fields, then update the _records annotation in the agent initialization to use list of that TypedDict instead of list[dict].Source: Coding guidelines
plugins/nemo-experimentalist/examples/smoke-agent/agent/harbor_wrapper.py (2)
87-101: 🔒 Security & Privacy | 🟠 Major | ⚡ Quick win
Filter excluded paths recursively.
Line 87 filters only top-level entries. Lines 100-101 upload nested
.env,.venv, and other excluded paths inside a selected directory. A nested credential file can reach the task container.Build an allowlisted staging directory, or apply the exclusions to every descendant before upload.
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@plugins/nemo-experimentalist/examples/smoke-agent/agent/harbor_wrapper.py` around lines 87 - 101, The upload flow around _reject_symlinks and the selected-entry loop must apply EXCLUDE and EXCLUDE_GLOB recursively, not only to AGENT_DIR’s top-level children. Build an allowlisted staging tree or otherwise ensure every descendant is filtered before environment.upload_dir or environment.upload_file, preventing nested credentials and excluded directories such as .env and .venv from reaching the container.
112-115: 📐 Maintainability & Code Quality | 🟡 Minor | ⚡ Quick win
🧩 Analysis chain
🏁 Script executed:
#!/bin/bash set -eu printf '%s\n' '--- target context ---' sed -n '90,130p' plugins/nemo-experimentalist/examples/smoke-agent/agent/harbor_wrapper.py printf '%s\n' '--- relevant uv and Python execution guidance ---' rg -n -i --glob 'README*' --glob '*.md' --glob '*.py' --glob '*.yaml' --glob '*.yml' --glob '*.toml' \ 'uv run|python main\.py|nooa|smoke-agent|task image' \ plugins/nemo-experimentalist . 2>/dev/null | head -n 240 printf '%s\n' '--- smoke-agent files ---' git ls-files plugins/nemo-experimentalist/examples/smoke-agentRepository: NVIDIA-NeMo/nemo-platform
Length of output: 50381
🏁 Script executed:
#!/bin/bash set -eu for f in \ plugins/nemo-experimentalist/examples/smoke-agent/dataset/_shared/Dockerfile \ plugins/nemo-experimentalist/examples/smoke-agent/AGENT-SPEC.md \ plugins/nemo-experimentalist/examples/smoke-agent/README.md \ plugins/nemo-experimentalist/examples/smoke-agent/agent/main.py \ plugins/nemo-experimentalist/examples/smoke-agent/agent/harbor_wrapper.py \ plugins/nemo-experimentalist/examples/tau3-nooa-agent/harbor_wrapper.py \ plugins/nemo-experimentalist/examples/tau3-nooa-agent/README.md do printf '\n--- %s ---\n' "$f" wc -l "$f" case "$f" in *Dockerfile) cat -n "$f" ;; *AGENT-SPEC.md) cat -n "$f" ;; *smoke-agent/README.md) sed -n '1,115p' "$f"; sed -n '180,215p' "$f" ;; *smoke-agent/agent/main.py) cat -n "$f" ;; *smoke-agent/agent/harbor_wrapper.py) cat -n "$f" ;; *tau3-nooa-agent/harbor_wrapper.py) sed -n '55,155p' "$f" ;; *tau3-nooa-agent/README.md) cat -n "$f" ;; esac doneRepository: NVIDIA-NeMo/nemo-platform
Length of output: 25677
🏁 Script executed:
#!/bin/bash set -eu printf '%s\n' '--- smoke-agent tests concerning the image and wrapper ---' for f in \ plugins/nemo-experimentalist/tests/experimentalist/test_smoke_agent_assets.py \ plugins/nemo-experimentalist/tests/experimentalist/test_smoke_agent.py \ plugins/nemo-experimentalist/tests/experimentalist/test_smoke_agent_baseline.py \ plugins/nemo-experimentalist/tests/experimentalist/test_smoke_agent_traces.py do printf '\n--- %s ---\n' "$f" rg -n -C 5 'Dockerfile|uv|python main|harbor|NOOA|nooa|image|wrapper|exec' "$f" || true done printf '%s\n' '--- image and dependency declarations ---' rg -n -C 4 'FROM |COPY --from|uv |nooa|pyproject|uv.lock|docker_image' \ plugins/nemo-experimentalist/examples/smoke-agent \ plugins/nemo-experimentalist/pyproject.toml \ pyproject.toml printf '%s\n' '--- available uv executable and run semantics ---' if command -v uv >/dev/null 2>&1; then uv --version uv run --help | sed -n '1,180p' else echo 'uv is not available in the verification sandbox' fiRepository: NVIDIA-NeMo/nemo-platform
Length of output: 50383
🌐 Web query:
uv run behavior without pyproject.toml project-less directory system site-packages💡 Result:
When you use uv run in a directory without a pyproject.toml file, uv operates in a project-less context [1]. Its behavior is determined by the following rules: 1. Project Discovery: Without a pyproject.toml, uv does not treat the directory as a project [1][2]. If you have a.venv directory in your current or parent directory, uv will detect and use that virtual environment [2]. If no virtual environment is found, uv will fall back to using the discovered Python interpreter [1][2]. 2. Handling System Site-Packages: uv creates isolated environments and does not natively support mixing system-installed packages with packages installed inside a uv-managed environment [3][4]. While you can create a virtual environment with the --system-site-packages flag (e.g., uv venv --system-site-packages), this only grants the environment access to system packages at runtime [5][4]. Crucially, uv does not consider these system packages when calculating dependencies, installing packages, or resolving environments [5][4]. This means tools like uv pip list or uv add will remain unaware of the system-installed packages [5][4]. 3. Common Pitfalls: If you have a pyproject.toml that lacks a [project] table, uv may still attempt to treat it as a project and fail with an error [6]. In such cases, or if you simply wish to avoid project discovery entirely, you can use the --no-project flag to force uv to ignore any existing project configuration and run the command in an isolated or environment-agnostic context [1][7][6]. In summary, uv is designed to maintain strict isolation [3]. If you have a requirement to use system-level packages alongside uv-managed dependencies, the recommended approach is to have uv manage all dependencies directly, as it does not support "inheriting" or mixing in system site-packages for its dependency management workflows [3][5].
Citations:
- 1: https://pydevtools.com/handbook/explanation/what-happens-when-you-run-uv-run/
- 2: https://docs.astral.sh/uv/reference/cli/
- 3: https://stackoverflow.com/questions/79839386/use-system-packages-inside-a-python-script-with-uv
- 4: astral-sh/uv#4466
- 5: astral-sh/uv#17215
- 6: astral-sh/uv#7068
- 7: https://github.com/astral-sh/uv/blob/262a50bb/docs/guides/scripts.md
🌐 Web query:
site:docs.astral.sh/uv uv run --no-project --frozen no pyproject.toml behavior💡 Result:
When you use uv run with the --no-project and --frozen flags, the behavior is as follows: --no-project: This flag prevents uv from discovering or using any project or workspace in the current directory or its parents [1][2]. Instead of using the project environment (the.venv created by the pyproject.toml), uv will create an isolated, ephemeral environment [1][2]. This means it will not install your current project, nor will it incorporate dependencies defined in your pyproject.toml [1][3][2]. --frozen: When used in conjunction with other flags or within a project, this tells uv to use the lockfile (if present) as the source of truth without attempting to update it [1][4]. However, when combined with --no-project, you are typically creating an isolated environment from scratch (e.g., using --with to specify dependencies) [1][2]. If you are not providing any dependencies via --with, and there is no project environment being used, uv will operate in a minimal environment [1][2]. If the command you run relies on packages, they must be specified via --with, or they must already be present in the environment (which is generally not the case for ephemeral --no-project environments unless explicitly handled) [1][2]. In summary, --no-project tells uv to ignore the pyproject.toml and project-related environment management entirely, opting for an isolated, ephemeral environment, while --frozen restricts uv from modifying or attempting to update any lockfiles [1][4][2]. If you use both, you are effectively running a command in a highly restricted, non-project-aware state that will not touch your project files [1][2].
Citations:
- 1: https://docs.astral.sh/uv/reference/cli/
- 2: https://docs.astral.sh/uv/reference/cli/index.md
- 3: https://docs.astral.sh/uv/guides/scripts/
- 4: https://docs.astral.sh/uv/concepts/projects/sync/
Run
main.pythroughuv.Use
uv run python main.py .... The task image includesuvand installs NOOA into its system interpreter. Do not use--no-projector--frozen, which would not reuse that installation.🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@plugins/nemo-experimentalist/examples/smoke-agent/agent/harbor_wrapper.py` around lines 112 - 115, Update the command constructed in the environment.exec call to invoke main.py through uv using “uv run python main.py”, while preserving the existing quoted instruction and session_id arguments. Do not add --no-project or --frozen.Source: Coding guidelines
plugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g2-name-patterns/validation/lookup-ann-marie/instruction.md (1)
1-9: 🔒 Security & Privacy | 🟠 Major | 🏗️ Heavy lift
🧩 Analysis chain
🏁 Script executed:
#!/bin/bash set -euo pipefail files=( plugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g2-name-patterns/validation/lookup-ann-marie/instruction.md plugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g2-name-patterns/validation/lookup-grace/instruction.md plugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g2-name-patterns/validation/lookup-role-obrien/instruction.md plugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g3-long-inputs/train/plain-dept/instruction.md plugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g3-long-inputs/validation/trailing-prose/instruction.md ) for file in "${files[@]}"; do grep -q 'SPDX-License-Identifier: Apache-2.0' "$file" || { echo "Missing Apache-2.0 identifier: $file" exit 1 } done rg -n 'SPDX|Apache-2.0|fixture|license' plugins/nemo-experimentalistRepository: NVIDIA-NeMo/nemo-platform
Length of output: 319
🏁 Script executed:
#!/bin/bash set -euo pipefail files=( plugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g2-name-patterns/validation/lookup-ann-marie/instruction.md plugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g2-name-patterns/validation/lookup-grace/instruction.md plugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g2-name-patterns/validation/lookup-role-obrien/instruction.md plugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g3-long-inputs/train/plain-dept/instruction.md plugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g3-long-inputs/validation/trailing-prose/instruction.md ) printf '%s\n' '--- target file metadata and contents ---' for file in "${files[@]}"; do printf '\n### %s\n' "$file" wc -l "$file" cat -n "$file" done printf '%s\n' '--- repository policy and fixture metadata references ---' rg -n -i --glob '!**/.venv/**' \ 'SPDX|Apache-2\.0|fixture|metadata|smoke-agent|instruction\.md|license' \ plugins/nemo-experimentalist .github README.md 2>/dev/null | head -n 400 printf '%s\n' '--- nearby and representative files ---' find plugins/nemo-experimentalist -type f \( -name '*.md' -o -name '*.yaml' -o -name '*.yml' -o -name '*.json' -o -name '*.py' -o -name '*.toml' \) \ | sort | head -n 250Repository: NVIDIA-NeMo/nemo-platform
Length of output: 50383
🏁 Script executed:
#!/bin/bash set -euo pipefail printf '%s\n' '--- scoped policy ---' cat -n plugins/nemo-experimentalist/AGENTS.md | sed -n '1,100p' printf '%s\n' '--- smoke-agent files and references ---' find plugins/nemo-experimentalist/examples/smoke-agent -type f -print | sort rg -n -F 'smoke-agent' plugins/nemo-experimentalist --glob '*.py' --glob '*.yaml' --glob '*.yml' --glob '*.toml' --glob '*.md' rg -n -F 'instruction.md' plugins/nemo-experimentalist --glob '*.py' --glob '*.yaml' --glob '*.yml' --glob '*.toml' rg -n -i 'read_text|instruction|prompt|dataset' plugins/nemo-experimentalist/src plugins/nemo-experimentalist/benchmarks --glob '*.py' | head -n 300 printf '%s\n' '--- SPDX coverage under the example tree ---' python3 - <<'PY' from pathlib import Path root = Path("plugins/nemo-experimentalist/examples/smoke-agent") for path in sorted(p for p in root.rglob("*") if p.is_file()): text = path.read_text(encoding="utf-8", errors="replace") if "SPDX-License-Identifier: Apache-2.0" not in text: print(path) PYRepository: NVIDIA-NeMo/nemo-platform
Length of output: 50381
🏁 Script executed:
#!/bin/bash set -euo pipefail files=( plugins/nemo-experimentalist/examples/smoke-agent/AGENT-SPEC.md plugins/nemo-experimentalist/examples/smoke-agent/README.md plugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g2-name-patterns/validation/lookup-ann-marie/task.toml plugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g3-long-inputs/validation/trailing-prose/task.toml plugins/nemo-experimentalist/tests/experimentalist/test_smoke_agent_assets.py plugins/nemo-experimentalist/tests/experimentalist/test_smoke_agent_gate.py plugins/nemo-experimentalist/tests/experimentalist/test_smoke_agent_baseline.py ) for file in "${files[@]}"; do printf '\n--- %s ---\n' "$file" wc -l "$file" case "$file" in *.py) rg -n -C 8 'instruction\.md|read_text|SPDX|digest|hash|task\.toml|asset' "$file" ;; *) cat -n "$file" | sed -n '1,220p' ;; esac done printf '\n--- all metadata-like files in smoke-agent ---\n' find plugins/nemo-experimentalist/examples/smoke-agent -type f \ \( -iname '*meta*' -o -iname '*license*' -o -iname '*.json' -o -iname '*.toml' -o -iname '*.yaml' -o -iname '*.yml' \) \ -print | sortRepository: NVIDIA-NeMo/nemo-platform
Length of output: 40237
🏁 Script executed:
#!/bin/bash set -euo pipefail python3 - <<'PY' from pathlib import Path targets = [ Path("plugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g2-name-patterns/validation/lookup-ann-marie/instruction.md"), Path("plugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g2-name-patterns/validation/lookup-grace/instruction.md"), Path("plugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g2-name-patterns/validation/lookup-role-obrien/instruction.md"), Path("plugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g3-long-inputs/train/plain-dept/instruction.md"), Path("plugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g3-long-inputs/validation/trailing-prose/instruction.md"), ] assert len(targets) == 5 for path in targets: text = path.read_text(encoding="utf-8") assert "SPDX-License-Identifier: Apache-2.0" not in text, path assert "SPDX-FileCopyrightText:" not in text, path assert text == text.strip() + "\n", path assert (path.parent / "task.toml").is_file(), path gate = Path("plugins/nemo-experimentalist/tests/experimentalist/test_smoke_agent_gate.py").read_text(encoding="utf-8") baseline = Path("plugins/nemo-experimentalist/tests/experimentalist/test_smoke_agent_baseline.py").read_text(encoding="utf-8") assets = Path("plugins/nemo-experimentalist/tests/experimentalist/test_smoke_agent_assets.py").read_text(encoding="utf-8") assert '(task_dir / "instruction.md").read_text(encoding="utf-8").strip()' in gate assert '(task / "instruction.md").read_text(encoding="utf-8").strip()' in baseline assert 'for name in ("instruction.md", "task.toml", "tests/expected.txt")' in assets assert "fixture exception" not in gate.lower() + baseline.lower() + assets.lower() assert "metadata mechanism" not in gate.lower() + baseline.lower() + assets.lower() header = ( "<!-- SPDX-FileCopyrightText: Copyright (c) 2025-2026 NVIDIA CORPORATION " "& AFFILIATES. All rights reserved. -->\n" "<!-- SPDX-License-Identifier: Apache-2.0 -->\n\n" ) for path in targets: original = path.read_text(encoding="utf-8") assert (header + original).strip() != original.strip(), path assert (header + original).startswith(header), path print("targets_missing_both_headers=5") print("approved_fixture_exception_or_metadata_path=absent") print("instruction_content_is_read_directly= true") print("literal_header_changes_prompt_bytes= true")Repository: NVIDIA-NeMo/nemo-platform
Length of output: 417
Define a non-semantic SPDX mechanism for the five prompt fixtures.
All five files lack both required SPDX lines. Do not prepend literal headers: the loader reads
instruction.mddirectly, so headers change prompt content and baseline behavior. Store metadata outside the prompt or strip a defined metadata block before loading, with tests.🧰 Tools
🪛 markdownlint-cli2 (0.23.2)
[warning] 1-1: First line in a file should be a top-level heading
(MD041, first-line-heading, first-line-h1)
[warning] 5-5: Fenced code blocks should have a language specified
(MD040, fenced-code-language)
📍 Affects 5 files
plugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g2-name-patterns/validation/lookup-ann-marie/instruction.md#L1-L9(this comment)plugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g2-name-patterns/validation/lookup-grace/instruction.md#L1-L9plugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g2-name-patterns/validation/lookup-role-obrien/instruction.md#L1-L9plugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g3-long-inputs/train/plain-dept/instruction.md#L1-L9plugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g3-long-inputs/validation/trailing-prose/instruction.md#L1-L9🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@plugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g2-name-patterns/validation/lookup-ann-marie/instruction.md` around lines 1 - 9, Define a non-semantic SPDX metadata mechanism for all five instruction.md fixtures without adding literal headers to prompt content: plugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g2-name-patterns/validation/lookup-ann-marie/instruction.md (lines 1-9), lookup-grace/instruction.md (lines 1-9), lookup-role-obrien/instruction.md (lines 1-9), plugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g3-long-inputs/train/plain-dept/instruction.md (lines 1-9), and validation/trailing-prose/instruction.md (lines 1-9). Store the required SPDX lines outside the prompts or implement a defined metadata-block stripping path in the instruction loader, and add tests confirming the loaded prompt content and baseline behavior remain unchanged.Source: Coding guidelines
plugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g4-dispatch-order/train/count-research/instruction.md (1)
1-1: 📐 Maintainability & Code Quality | 🟠 Major | ⚡ Quick win
Add required SPDX metadata to every task instruction.
These files lack the required NVIDIA copyright and Apache-2.0 identifiers. Use Markdown comments. Preserve the prompt content and validate task behavior after the change.
plugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g4-dispatch-order/train/count-research/instruction.md#L1-L1: add SPDX comments before the instruction.plugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g4-dispatch-order/train/lookup-ada/instruction.md#L1-L1: add SPDX comments before the instruction.plugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g4-dispatch-order/validation/count-engineers-research/instruction.md#L1-L1: add SPDX comments before the instruction.plugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g4-dispatch-order/validation/count-operators-ops/instruction.md#L1-L1: add SPDX comments before the instruction.plugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g4-dispatch-order/validation/lookup-grace/instruction.md#L1-L1: add SPDX comments before the instruction.plugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g5-edge-cases/train/empty-role/instruction.md#L1-L1: add SPDX comments before the instruction.🧰 Tools
🪛 markdownlint-cli2 (0.23.2)
[warning] 1-1: First line in a file should be a top-level heading
(MD041, first-line-heading, first-line-h1)
📍 Affects 6 files
plugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g4-dispatch-order/train/count-research/instruction.md#L1-L1(this comment)plugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g4-dispatch-order/train/lookup-ada/instruction.md#L1-L1plugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g4-dispatch-order/validation/count-engineers-research/instruction.md#L1-L1plugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g4-dispatch-order/validation/count-operators-ops/instruction.md#L1-L1plugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g4-dispatch-order/validation/lookup-grace/instruction.md#L1-L1plugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g5-edge-cases/train/empty-role/instruction.md#L1-L1🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@plugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g4-dispatch-order/train/count-research/instruction.md` at line 1, Add Markdown SPDX copyright and Apache-2.0 license comments before the existing prompt in each listed instruction.md file: plugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g4-dispatch-order/train/count-research/instruction.md (lines 1-1), plugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g4-dispatch-order/train/lookup-ada/instruction.md (lines 1-1), plugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g4-dispatch-order/validation/count-engineers-research/instruction.md (lines 1-1), plugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g4-dispatch-order/validation/count-operators-ops/instruction.md (lines 1-1), plugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g4-dispatch-order/validation/lookup-grace/instruction.md (lines 1-1), and plugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g5-edge-cases/train/empty-role/instruction.md (lines 1-1). Preserve each instruction’s prompt content exactly and validate task behavior after adding the metadata.Source: Coding guidelines
plugins/nemo-experimentalist/examples/smoke-agent/dataset/task-template/instruction.md (1)
1-9: 📐 Maintainability & Code Quality | 🟠 Major | 🏗️ Heavy lift
Add the required SPDX identifiers to each task fixture.
Use non-rendered Markdown comments. Preserve the task prompt and output format.
plugins/nemo-experimentalist/examples/smoke-agent/dataset/task-template/instruction.md#L1-L9: Add the NVIDIA copyright and Apache-2.0 SPDX identifiers.plugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g1-aggregation/validation/total-hours-analysts/instruction.md#L1-L9: Add the NVIDIA copyright and Apache-2.0 SPDX identifiers.plugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g1-aggregation/validation/total-hours-ops/instruction.md#L1-L9: Add the NVIDIA copyright and Apache-2.0 SPDX identifiers.plugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g5-edge-cases/train/missing-person/instruction.md#L1-L9: Add the NVIDIA copyright and Apache-2.0 SPDX identifiers.plugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g5-edge-cases/validation/lookup-grace/instruction.md#L1-L9: Add the NVIDIA copyright and Apache-2.0 SPDX identifiers.plugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g5-edge-cases/validation/missing-person-hours/instruction.md#L1-L9: Add the NVIDIA copyright and Apache-2.0 SPDX identifiers.plugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g5-edge-cases/validation/missing-person-role/instruction.md#L1-L9: Add the NVIDIA copyright and Apache-2.0 SPDX identifiers.🧰 Tools
🪛 markdownlint-cli2 (0.23.2)
[warning] 1-1: First line in a file should be a top-level heading
(MD041, first-line-heading, first-line-h1)
[warning] 5-5: Fenced code blocks should have a language specified
(MD040, fenced-code-language)
📍 Affects 7 files
plugins/nemo-experimentalist/examples/smoke-agent/dataset/task-template/instruction.md#L1-L9(this comment)plugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g1-aggregation/validation/total-hours-analysts/instruction.md#L1-L9plugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g1-aggregation/validation/total-hours-ops/instruction.md#L1-L9plugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g5-edge-cases/train/missing-person/instruction.md#L1-L9plugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g5-edge-cases/validation/lookup-grace/instruction.md#L1-L9plugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g5-edge-cases/validation/missing-person-hours/instruction.md#L1-L9plugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g5-edge-cases/validation/missing-person-role/instruction.md#L1-L9🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@plugins/nemo-experimentalist/examples/smoke-agent/dataset/task-template/instruction.md` around lines 1 - 9, Add non-rendered NVIDIA copyright and Apache-2.0 SPDX identifier comments to each affected instruction.md fixture, preserving every existing task prompt and required output format: plugins/nemo-experimentalist/examples/smoke-agent/dataset/task-template/instruction.md lines 1-9; plugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g1-aggregation/validation/total-hours-analysts/instruction.md lines 1-9; plugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g1-aggregation/validation/total-hours-ops/instruction.md lines 1-9; plugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g5-edge-cases/train/missing-person/instruction.md lines 1-9; plugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g5-edge-cases/validation/lookup-grace/instruction.md lines 1-9; plugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g5-edge-cases/validation/missing-person-hours/instruction.md lines 1-9; and plugins/nemo-experimentalist/examples/smoke-agent/dataset/groups/g5-edge-cases/validation/missing-person-role/instruction.md lines 1-9.Source: Coding guidelines
plugins/nemo-experimentalist/examples/smoke-agent/README.md (1)
199-202: 📐 Maintainability & Code Quality | 🟡 Minor | ⚡ Quick win
Correct the gate assertion count.
The PR adds an eight-assertion gate. This section says five assertions and omits failed-generalization candidate rejection.
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@plugins/nemo-experimentalist/examples/smoke-agent/README.md` around lines 199 - 202, Update the gate description near the existing “Five assertions” text to state that it contains eight assertions, including rejection of candidates that fail generalization on held-out instances. Ensure the listed assertion categories accurately cover all eight checks.plugins/nemo-experimentalist/examples/smoke-agent/scripts/build_all_group.py (1)
64-68: 🗄️ Data Integrity & Integration | 🟠 Major | ⚡ Quick win
Do not suppress generated-directory deletion errors.
Line 67 ignores a failed deletion. A partial deletion can retain tasks from a removed or excluded group, then mix them with current tasks in
_all.Proposed fix
- shutil.rmtree(target, ignore_errors=True) + if target.exists(): + shutil.rmtree(target)Based on learnings: the combined-group test requires generated tasks to exactly match their sources.
📝 Committable suggestion
‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.def assemble(dataset_dir: Path) -> list[Path]: """Rebuild the combined group from its sources; return the task directories written.""" target = dataset_dir / "groups" / COMBINED if target.exists(): shutil.rmtree(target)🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@plugins/nemo-experimentalist/examples/smoke-agent/scripts/build_all_group.py` around lines 64 - 68, Update assemble so removal of the generated target directory does not suppress deletion errors: replace the ignore_errors behavior in the shutil.rmtree call with error propagation. Keep the existing target path and rebuild flow unchanged so assembly stops instead of continuing with stale contents when deletion is incomplete.plugins/nemo-experimentalist/src/nemo_experimentalist_plugin/experimentalist/components/proposer.py (1)
163-170: 🎯 Functional Correctness | 🟠 Major | ⚡ Quick win
Make repeated types permissible in the proposer prompt.
Lines 163-170 allow reused types.
_run_with_contextstill requiresavailable_typesand avoidstried_types. The model will not propose a reused type while an untried type exists. Change the prompt to preferavailable_types, not require them. Add a prompt contract test.Proposed fix
- - tried_types (list[str]): optimization_types already attempted — AVOID these - - available_types (list[str]): types not yet tried — PICK FROM THESE + - tried_types (list[str]): optimization_types already attempted — reuse when analysis supports it + - available_types (list[str]): types not yet tried — prefer these for novelty ... - 3. Load the matching card with `doc(self.optimize.<name>)` and pick ONE - optimization_type from `available_types` that the card covers. + 3. Load the matching card with `doc(self.optimize.<name>)` and pick ONE + optimization_type from `available_types` or `tried_types` that the card covers. ... - - Each Improvement.optimization_type MUST be in `available_types`. If - `available_types` is empty, pick the least-tried type from the tried set. + - Prefer an Improvement.optimization_type in `available_types`. Reuse a + type from `tried_types` when it is the best match for the root cause.Based on supplied change details, novelty is intended to be a preference, not a requirement.
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@plugins/nemo-experimentalist/src/nemo_experimentalist_plugin/experimentalist/components/proposer.py` around lines 163 - 170, Update the proposer prompt construction in _run_with_context so available_types are presented as a preference rather than a requirement, allowing types from tried_types to be proposed when appropriate. Preserve the all_types allowance used by _filter_improvements, and add a prompt contract test verifying reused types remain permissible while untried types are favored.plugins/nemo-experimentalist/tests/experimentalist/test_smoke_agent_assets.py (1)
39-41: 🎯 Functional Correctness | 🟠 Major | ⚡ Quick win
Fail when the fixture has no tasks.
An absent
dataset/groupsdirectory returns an empty list. The image and verifier tests then return, and the environment test iterates zero tasks. Deleting all task assets can pass this gate.Assert that the groups directory and task list exist in
_task_tomls(). Remove the early returns that depend on an empty task list. Keep_alloptional in its separate tests.Proposed fix
def _task_tomls() -> list[Path]: groups = _EXAMPLE_DIR / "dataset" / "groups" - return sorted(groups.rglob("task.toml")) if groups.is_dir() else [] + assert groups.is_dir(), f"missing smoke-agent task groups: {groups}" + tasks = sorted(groups.rglob("task.toml")) + assert tasks, f"no smoke-agent tasks found under {groups}" + return tasksAlso applies to: 145-147, 159-161, 179-186
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@plugins/nemo-experimentalist/tests/experimentalist/test_smoke_agent_assets.py` around lines 39 - 41, Update `_task_tomls()` to assert that `dataset/groups` exists and that its recursive `task.toml` result is non-empty, failing when task fixtures are missing. Remove early returns in the image, verifier, and environment tests that skip execution for an empty task list, while preserving `_all` as optional in its separate tests.plugins/nemo-experimentalist/tests/experimentalist/test_smoke_agent_gate.py (1)
108-127: 🔒 Security & Privacy | 🔴 Critical | 🏗️ Heavy lift
Do not execute candidate code in the pytest process.
spec.loader.exec_module(module)executesagent.pyfromSMOKE_EXPERIMENT_DIR. Candidate code is generated during optimization and can run arbitrary host commands, read CI credentials, or modify the checkout during this test.Replay candidates only in the existing isolated Harbor or container environment. A host subprocess is not an isolation boundary.
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@plugins/nemo-experimentalist/tests/experimentalist/test_smoke_agent_gate.py` around lines 108 - 127, Update _agent_class so candidate agent.py files are never imported or executed in the pytest process. Remove the direct importlib execution path and replay each candidate through the existing isolated Harbor or container environment, returning its ReportAgent-compatible result via that boundary; do not use a host subprocess as a substitute for isolation.plugins/nemo-experimentalist/tests/experimentalist/test_trace_scorer_contract.py (1)
43-53: 🎯 Functional Correctness | 🟡 Minor | ⚡ Quick win
Assert the complete zero-turn grounding rule.
The test accepts a contract that only says “call graph.” It does not require methods, execution order, status, or the prohibition on retrying turn lookups. A regression that removes those requirements still passes.
Assert
"name the methods that ran","the order they ran in","their status", and"do not retry".🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@plugins/nemo-experimentalist/tests/experimentalist/test_trace_scorer_contract.py` around lines 43 - 53, Strengthen test_contract_offers_an_alternative_grounding by asserting the contract includes “name the methods that ran”, “the order they ran in”, “their status”, and “do not retry” in addition to the existing call-graph requirement. Keep the zero-turn grounding and forbidden-ID assertions unchanged.
… selects it `configs/generalization.yaml` and `configs/repair.yaml` are identical apart from their comments, and `optimizer.yaml` pins both dataset splits to g1-aggregation. Running the documented command with `--config configs/generalization.yaml` therefore ran the repair scenario against g1 and quietly answered the wrong question. The generalization scenario only ever ran because the g4 splits were passed on the command line, which the README did not show. The README also claimed the two configs "differ in one line, max_train_batch_tasks". That was true when scenario class was a sampling decision; it stopped being true when the class moved to being a property of the split, and the sentence was never updated. Adds optimizer-generalization.yaml, identical to optimizer.yaml except that it points at g4-dispatch-order. The profile is the right home for this: which split a run uses is what decides whether the winner should beat the baseline or the baseline should be retained, and that is profile-level information rather than tuning. The scenario table now names a profile per row. Also records why the task fixtures carry no licence header, so the question does not get reopened: instruction.md is the prompt and expected.txt is compared byte-for-byte, so a header changes what the task measures. This matches the repository as it stands -- the copyright-fix hook is scoped to .py and .ts, and comparable fixtures elsewhere carry no header either. Signed-off-by: Christian Schüller <cschueller@nvidia.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Why
The existing example agents take over two hours per run, and their outcomes are stochastic — so they can confirm the loop completed, but never that it improved anything. That makes them unusable as a feedback loop while refactoring the Experimentalist.
This adds a fixture that answers a stronger question: did the Experimentalist actually repair a weakness we know the agent has?
How it works
The agent under test is a NOOA agent that makes no model calls — handlers are regular expressions plus a dict lookup — so the only stochastic component in a run is the Experimentalist itself. NOOA still instruments plain methods, so traces carry the real call graph with each method's input and output. Its completion client points at an unroutable address, which turns an accidental model call into a loud failure rather than a quiet loss of determinism.
The agent is never patched. It ships five known weaknesses, and a scenario is a different task set pointed at the same agent — which mirrors how optimization is actually used ("here is my agent, here is a benchmark, find what is wrong"). Each group holds two failing tasks and one control per split, so a real fix is distinguishable from a special case and a destructive one costs reward.
Two scenario classes, with opposite pass conditions
Both demonstrated against real runs:
repairgeneralizationThe distinction is which tasks sit in train, not a config knob. Running one while expecting the other makes a result unreadable, so the gate is scenario-aware.
The gate
Eight assertions covering the ways a result can look better than it is: the winner's code actually changed, reward improved beyond noise, controls still pass, the analysis named the cause, held-out instances are handled, and — for the generalization class — that a candidate genuinely fixed train before being rejected.
Each was verified to fail by mutating a copy of a real experiment. An assertion that cannot fail is not a gate.
Cost
Evaluation is negligible by design. The cost is the Experimentalist's own components, which are the part under test.
Notes for reviewers
🤖 Generated with Claude Code
Summary by CodeRabbit
New Features
Documentation
Bug Fixes
Tests