feat(llmobs): add experiment multirun concurrency - #9686
Conversation
Overall package sizeSelf size: 7.61 MB Dependency sizes| name | version | self size | total size | |------|---------|-----------|------------| | import-in-the-middle | 3.3.3 | 125.43 kB | 441.68 kB | | opentracing | 0.14.7 | 194.81 kB | 194.81 kB | | dc-polyfill | 0.1.11 | 25.74 kB | 25.74 kB |🤖 This report was automatically generated by heaviest-objects-in-the-universe |
|
Codecov Report❌ Patch coverage is
Additional details and impacted files@@ Coverage Diff @@
## feat/llmobs-experiment-recorder-stacked #9686 +/- ##
===========================================================================
- Coverage 98.51% 89.62% -8.90%
===========================================================================
Files 953 931 -22
Lines 132199 128403 -3796
Branches 11447 10209 -1238
===========================================================================
- Hits 130237 115078 -15159
- Misses 1962 13325 +11363 Flags with carried forward coverage won't be shown. Click here to find out more. ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
| usesLLMObsTrace, | ||
| }) | ||
|
|
||
| const timestampMs = Date.now() |
There was a problem hiding this comment.
did this move into runSingle
There was a problem hiding this comment.
| }) | ||
| } | ||
|
|
||
| const evaluatorResults = await Promise.all(pending) |
There was a problem hiding this comment.
are we parallelizing eval computation across multiple evals within a single record? and limit applies at the record level?
There was a problem hiding this comment.
yep exactly, i think we might have to adjust this when we introduce llmaj evals as a class, so users don't get rate limited
There was a problem hiding this comment.
i think this approach is fine for am initial version, we can adjust if we get feedback.
IS this how the python version works or that one applies a more global concept of rate limiting?
2 participants
What does this PR do?
Adds local LLMObs experiment multirun and concurrency support for the Node.js SDK:
runstotracer.llmobs.experiments.experiment({ ... })for local experiments..run({ concurrency })to bound concurrent task/evaluator work within each run.10, matching the Python SDK.result.runs[], withresult.rowsandresult.summaryEvaluationskept as aliases to the first run for compatibility.run_count, UUIDrun_id, andrun_iterationmetadata/tags for backend events and metrics.Motivation
Bring the Node.js experiments API closer to Python SDK behavior and support Omnea-requested local experiment workflows where each dataset row can be evaluated across multiple sequential runs, with bounded parallelism for tasks and evaluators.
Additional Notes
This PR is stacked on
feat/llmobs-experiment-recorder-stacked.run_idgeneration usesnode:crypto.randomUUID()via the experiments util helper to match Python's UUID v4 run IDs.Validation:
./node_modules/.bin/mocha packages/dd-trace/test/llmobs/experiments/experiment.spec.js packages/dd-trace/test/llmobs/experiments/index.spec.js packages/dd-trace/test/llmobs/experiments/util.spec.js— 44 passing./node_modules/.bin/eslint packages/dd-trace/src/llmobs/experiments/experiment.js packages/dd-trace/src/llmobs/experiments/noop.js packages/dd-trace/src/llmobs/experiments/util.js packages/dd-trace/test/llmobs/experiments/experiment.spec.js packages/dd-trace/test/llmobs/experiments/index.spec.jsgit diff --checkEnd-to-end validation example: DataDog/llm-observability#90 adds a runnable Node.js multirun/concurrency multispan example in
experiments/nodejs/examples/04-multirun-concurrency.jsfor reviewers to validate this feature against these SDK changes.