docs(experimentalist): rewrite the plugin README for developers - #1165
Merged
Conversation
Contributor
|
Note Reviews pausedIt looks like this branch is under active development. To avoid overwhelming you with review comments due to an influx of new commits, CodeRabbit has automatically paused this review. You can configure this behavior by changing the Use the following commands to manage reviews:
Use the checkboxes below for quick actions:
📝 WalkthroughWalkthroughThe Experimentalist README now documents the optimization workflow, execution commands, deployment settings, tracing, sandbox isolation, artifacts, examples, benchmarks, and development commands. ChangesExperimentalist README
Possibly related PRs
Suggested reviewers: 🚥 Pre-merge checks | ✅ 5✅ Passed checks (5 passed)
✨ Finishing Touches🧪 Generate unit tests (beta)
Comment |
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
Contributor
There was a problem hiding this comment.
Actionable comments posted: 2
Caution
Some comments are outside the diff and can’t be posted inline due to platform limitations.
⚠️ Outside diff range comments (1)
plugins/nemo-experimentalist/README.md (1)
295-306: 🎯 Functional Correctness | 🟠 Major | ⚡ Quick winRemove
--workdir "$repo"fromsbx exec.
sbx execdoes not support--workdir, so this command fails. If a working directory is required, change to the clone path inside the sandbox;"$repo"is the read-only host source in clone mode.🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@plugins/nemo-experimentalist/README.md` around lines 295 - 306, Update the documented sbx exec command to remove the unsupported --workdir "$repo" option. If execution must occur in the cloned sandbox directory, change to that directory inside the sandbox and avoid using the read-only host source path represented by "$repo".
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@plugins/nemo-experimentalist/README.md`:
- Line 6: Add a Prerequisites section before the README overview, covering
checkout, uv, Docker, credentials, profile, and dataset requirements. Add a Next
Steps section after the development section with links to the workflow, sandbox,
and benchmark guides.
- Line 18: Refocus the README around the “How it works” explanation by moving
“Run it,” sandbox procedures, “Parameters,” and “Configuration” reference
material into linked documentation pages. Keep only explanatory content in this
page, and place optional trace or sandbox troubleshooting details in dropdowns
if they remain.
---
Outside diff comments:
In `@plugins/nemo-experimentalist/README.md`:
- Around line 295-306: Update the documented sbx exec command to remove the
unsupported --workdir "$repo" option. If execution must occur in the cloned
sandbox directory, change to that directory inside the sandbox and avoid using
the read-only host source path represented by "$repo".
🪄 Autofix
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Path: .coderabbit.yaml
Review profile: CHILL
Plan: Enterprise
Run ID: 3e0e2397-fcb4-4edc-868b-3945aea8ec99
📒 Files selected for processing (1)
plugins/nemo-experimentalist/README.md
Contributor
|
schuellc-nvidia
added a commit
that referenced
this pull request
Aug 7, 2026
The Configuration section documented `NEMO_EXPERIMENTALIST_API_BASE`, `_API_KEY`, and the smart/mid/fast tiers. #1159 deletes `settings.py` outright, so every one of those variables goes away: the optimizer now resolves a default/fast pair of Platform Model Entities selected by `nemo setup`, and reads no endpoint, provider key, or provider model name of its own. Rewrites the section against that PR's code rather than its prose, so the tier mapping is accurate -- the trajectory scorer moves from mid to fast, and the architecture doc from mid to default. Updates the sandbox `sbx exec --env` block to the two overrides that replace the six, and the secret-handling caution that assumed keys were among them. Also names the bounded sub-problem optimization loop in the round walkthrough. It was missing, and `implement` runs it before the integration check rather than inside it (coder.py:680-686). Review feedback from @gaiadilorenzo on #1165. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Signed-off-by: Christian Schüller <cschueller@nvidia.com>
schuellc-nvidia
added a commit
that referenced
this pull request
Aug 7, 2026
The six numbered steps describe a cycle, but read as a straight line: the loop back to Select, the drop path out of Implement, and the two places the two splits are consumed are all only implicit in the prose. Adds a mermaid flowchart above the list rather than replacing it -- the diagram carries the shape, the list keeps the detail that does not fit in a node. Syntax verified by parsing the extracted block with mermaid's own parser, and the constructs match those already rendering under docs/. Review feedback from @gaiadilorenzo on #1165. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Signed-off-by: Christian Schüller <cschueller@nvidia.com>
Contributor
There was a problem hiding this comment.
Actionable comments posted: 1
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@plugins/nemo-experimentalist/README.md`:
- Line 45: Update the optimizer diagram in README.md so the Base node connects
to a baseline validation step before Select, reflecting the initial agent-0
validation and clarifying that Select receives its input from that validation.
🪄 Autofix
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Path: .coderabbit.yaml
Review profile: CHILL
Plan: Enterprise
Run ID: c2df5189-5460-42da-a93e-52fb9ab45315
📒 Files selected for processing (1)
plugins/nemo-experimentalist/README.md
schuellc-nvidia
added a commit
that referenced
this pull request
Aug 7, 2026
The diagram ran Base straight into Select, which leaves the first selection with nothing to select on. `_run` evaluates agent-0 on validation as an explicit round-0 step before the cycle begins (loop.py:558), and the loop's own docstring records the order as baseline -> [convergence-check -> select -> ...]. Adds the step to the diagram and to the sentence above it, so the two agree. Mermaid re-parsed after the change. Review feedback from @coderabbitai on #1165. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Signed-off-by: Christian Schüller <cschueller@nvidia.com>
The README had grown into an operator runbook that duplicated the Experimentalist skill and the docs site while never explaining what the Experimentalist does. Refocus it on the plugin's function and link out for the rest. - Explain the optimization loop, the levers it can change, and the train-diagnoses / validation-selects split, matching the round order the implementation actually runs. - Add a parameter table built from the CLI options, plus smoke-first guidance, run cost and duration, and resume-on-rerun. - Document what ships in the box: the nooa and langchain-framework skills, the two example agents, and the canonical benchmark suites. - Lead with the example-agent walkthrough so readers start from a worked run. - Correct claims that did not hold against the source: the command is `nemo agents experimentalist doctor`; the evaluation container receives whatever each Harbor task's `task.toml` declares rather than a fixed forwarded set, with `AUT_MODEL_NAME` a convention of the example agents; NeMo Experiments receives a structure-only mirror; and the model table read as exhaustive when the fast model also drives sub-steps across most components. Keep the "Recommended laptop isolation" heading, which the Experimentalist skill links to by anchor, and carry forward the objective_function and regression_metrics reference from #1159 under a task-oriented heading. Signed-off-by: Christian Schüller <cschueller@nvidia.com>
schuellc-nvidia
force-pushed
the
improve-experimentalist-readme/cschueller
branch
from
August 8, 2026 00:14
4c64d5f to
f1996b6
Compare
schuellc-nvidia
enabled auto-merge
August 8, 2026 00:21
BrianNewsom
approved these changes
Aug 8, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
The
nemo-experimentalistREADME had grown into a 246-line operator runbook thatduplicated the Experimentalist skill and the docs site while never explaining what
the Experimentalist actually does — the evolutionary loop, the levers it changes,
and what the train and validation splits are each for. This rewrites it for human
developers: function first, links out for procedure, and a record of what the
plugin ships in the box.
Changes
mainand reconciled with feat(agents): allow optimization agents to configure models #1159, which rewrote themodel-configuration section while this branch was open. The
objective_function/regression_metricsreference from that PR is carriedforward under a task-oriented heading; the relaxed
uv >=0.9.14floor (build: allow newer uv for source checkouts #1146)and the
[tool.uv.sources]pointer (docs(experimentalist): point the nooa pin at the file that actually has it #1099) are carried forward too.under "Run it", so readers start from a worked run rather than a cold invocation.
(LLM call, deterministic code, tools, skills, structure, model/config), the
one-targeted-change-per-candidate rule, and the round order taken from
EvolutionaryOptimizer— including the train evaluation the old text omittedentirely. States plainly that train diagnoses and validation selects.
runcommand's Typer options, plusthe
doctorflags.tmux),smoke-first guidance, resume-on-rerun via the same
--experiment-dir, thatdoctorchecks neither datasets nor task template without a loaded profile,and that the
.git@marker is required to pin a ref.nooaandlangchain-frameworkframework skills,the two example agents, and the canonical benchmark suites.
nemo experimentalist doctor->nemo agents experimentalist doctor(the plugin registers undernemo.cli.agents), and astale note describing the pinned NOOA revision as "one past v0.0.6" when the
root
pyproject.tomlpins past v0.0.8 for a different reason.docs/agents/insight-driven-optimization.mdx.The
## Recommended laptop isolationheading is retained deliberately: theExperimentalist skill added in #1088 links to that anchor.
Type of Change
Quality Gates
Markdown file; no code, config, or generated output is touched.
Verification
Signed-off-by:traileruv run pre-commit run -apasses, or any blocked checks are identified belowTargeted validation:
uv run pre-commit run --files plugins/nemo-experimentalist/README.md— passed(merge-conflict check ran; all language-specific hooks correctly skipped).
uv run pre-commit run -a— passed. Note: theuv-lockhook re-resolveduv.lockin my local environment, dropping ppc64le/s390x wheel entries. That isunrelated environment drift, so it was reverted and is not part of this PR;
the only file changed is
plugins/nemo-experimentalist/README.md.Every factual claim added to the README was checked against the plugin source
rather than against the skill or the docs. Statements verified this way include
the CLI option list (
cli.py), the round order and resume behavior (loop.py),the six optimization cards (
cards.py), the one-change rule (proposer.py),the
storage.publish_winner: truedefault (config.py), the profile-gateddoctorchecks (cli.py), the.git@ref parsing (repository.py), and the89-task count for Terminal-Bench 2.1 (
benchmarks/suites/terminal-bench-2.1.yaml).All relative links in the file were confirmed to resolve.
A second verification pass against the source found three claims that did not hold,
all inherited from the previous README, now corrected:
The agent under test does not read a fixed forwarded set. What reaches the
evaluation container is declared per task in
task.toml;AUT_MODEL_NAMEis aconvention of the example agents and
benchmarks/run.py, not a plugin contract.NeMo Experiments receives a structure-only mirror —
experiment_mirror.pystatesthat rewards and trials are not copied into it.
The model-tier table read as exhaustive.
get_smart_modelalso drives theoptimizer loop, and
get_fast_modeldrives sub-steps inside most components.No test suite covers README prose, so none was run.