feat(agents): PerRequestUsageCapture capability on the after_model_request seam - #845
Open
mpfaffenberger wants to merge 2 commits into
Open
feat(agents): PerRequestUsageCapture capability on the after_model_request seam#845mpfaffenberger wants to merge 2 commits into
mpfaffenberger wants to merge 2 commits into
Conversation
…equest capability invoke_agent_with_model's per_request_usage and final_context_tokens were derived after the run by walking result.new_messages() for ModelResponse objects. Move the observation onto pydantic-ai's after_model_request capability seam: PerRequestUsageCapture records each response at the exact moment the run appends it to state (verified identical object for streamed and non-streamed requests on 2.31.0), and hands the invocation layer an owned capture at the run boundary. consume() is read-and-clear and double-gated: an identity gate rejects stale captures (streaming-retry re-entry, guest wrappers bypassing capabilities) and a consistency gate rejects captures whose response sequence no longer matches the result's recorded responses -- mid-run compaction can summarize away a long run's own earlier responses, and the eager walk never saw those. Every rejection falls back to the pre-capability new_messages() walk through the same extraction helpers, so both paths are byte-identical. Scope: run-total usage_metrics and duration_ms stay at the call site -- the timing spans streaming-retry attempts (multiple agent.run() calls) and cannot live inside a per-run capability with parity. The capability is only constructed when include_usage_metrics is set, mirroring the build_tool_output_limits conditional-splice pattern.
… order) Review pass 1 finding: CombinedCapability.after_model_request iterates reversed(self.capabilities), so last-in-list executes FIRST -- the previous comment had the onion inside out. Move the capture to first position so it executes last and records the response object that actually reaches run state even if another capability replaces it. Pin the semantics both ways: a replacer executing before the capture is observed and owned; the adversarial misordering is disowned by the consistency gate and falls back to the eager walk instead of reporting pre-replacement objects. Also: narrow the byte-identical claim to real AgentRunResult semantics (the consistency gate adds one new_messages() call that only a pathological stateful result object could observe), and modernize the typing spellings (list/tuple/X | None).
This was referenced Aug 22, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
PerRequestUsageCapture: sub-agent per-request usage as a capabilitySixteenth in the capability-conversion series (#828-#836, #838-#842, #844). First claim of the
after_model_requestseam.The feature
invoke_agent_with_modelreports token usage per model request (per_request_usage-- "use per_request_usage for pricing") and the final request's context size (final_context_tokens). On main, both are derived after the run by walkingresult.new_messages()forModelResponseobjects (code_puppy/tools/subagent_usage_metrics.py).The conversion
New
code_puppy/agents/_subagent_usage.py:PerRequestUsageCapture-- one instance per invocation.for_runhands eachagent.run()a fresh run-scoped collector (proven necessary: responses accumulate across runs on a shared instance, and streaming-retry re-entry means multiplerun()calls per invocation). The collector records eachModelResponseonafter_model_requestand stows(result, responses)into a one-slot capture atafter_run-- which only fires for runs that produced a result, so failed attempts never pollute the slot.consume(result)is read-and-clear and double-gated:captured result is result, feat(agents): promote run-end telemetry to a RunTelemetry capability (after_run seam) #844 precedent): rejects stale captures and guest wrappers that bypass capabilities.ModelResponses. This is not paranoia -- mid-run history rewrites are real: aProcessHistorypass that drops one of this run's own responses (the shape mid-run compaction produces on a long sub-agent run) leaves the seam having seen N responses whilenew_messages()records N-1. The eager walk on main never reported the dropped response; without the gate the capability would report a phantom usage entry. Pinned bytest_consistency_gate_rejects_mid_run_history_rewrite, which reproduces the divergence empirically.new_messages()walk -- and both paths feed the same extraction helpers (extract_per_request_usage/extract_final_context_tokens, untouched), so both paths are byte-identical.Seam mechanics (2.31.0, verified empirically)
after_model_requestfires once per model response, for both streamed and non-streamed requests (all three_finish_handlingcall sites), and receives the identicalModelResponseobject that_append_responsecommits to run state.CombinedCapabilityapplies after-hooks in reverse list order (onion semantics) -- caught by the reviewer in pass 1. The capture therefore goes first incapabilities=[...]: it executes last inafter_model_requestand records the response object that actually reaches run state even if another capability replaced it. Pinned both ways: the production ordering owns replacements; the adversarial misordering is disowned by the consistency gate and falls back rather than reporting pre-replacement objects.for_runisasyncin 2.31.0 and resolves once perrun();after_runfires on the resolved run capability with the identicalAgentRunResultthe caller receives (re-confirmed from feat(agents): promote run-end telemetry to a RunTelemetry capability (after_run seam) #844).Scope & bounded non-moves
usage_metrics(result.usage) andduration_msstay at the call site. The timing deliberately spans streaming-retry attempts -- multipleagent.run()calls -- so it cannot live inside a per-run capability with parity.include_usage_metricsis set (invoke_agent_with_modelonly); plaininvoke_agentinvocations get an empty splice and zero new machinery -- mirroring thebuild_tool_output_limitsconditional-splice pattern.subagent_usage_metrics.pyuntouched -- all 42 existing helper tests pass unchanged.Behavioral divergences
None for real
AgentRunResults (whosenew_messages()is a stable view over run state). The observation moved to the seam; the reported values are computed by the same helpers over the same objects, and every case where the seam data could differ from the eager walk is gated back onto the eager walk. The consistency gate adds onenew_messages()call, observable only by a pathological result object with statefulnew_messages()semantics.Tests
18 contract tests in
tests/agents/test_per_request_usage_capability.py: seam capture identity/order (streamed + non-streamed), read-and-clear, identity gate, consistency gate (incl. the mid-run rewrite reproduction and anew_messages()-raising result preserving the old failure mode), per-run isolation across sequential runs, extraction parity vs the eager walk (non-vacuous: FunctionModel reports real usage), construction helper, spec-constructibility, and a call-site wiring pin.Merge-order note
Same story as the rest of the series: this PR touches the sub-agent
capabilities=[...]block that #828-#842 also graze. Whoever lands last eats a trivial rebase. This one additionally shares the success-boundary region of_invoke_agent_implwith #842 (session persistence) -- also trivial.