Skip to content

feat(agents): PerRequestUsageCapture capability on the after_model_request seam - #845

Open
mpfaffenberger wants to merge 2 commits into
mainfrom
feature/per-request-usage-capability
Open

feat(agents): PerRequestUsageCapture capability on the after_model_request seam#845
mpfaffenberger wants to merge 2 commits into
mainfrom
feature/per-request-usage-capability

Conversation

@mpfaffenberger

@mpfaffenberger mpfaffenberger commented Aug 22, 2026

Copy link
Copy Markdown
Owner

PerRequestUsageCapture: sub-agent per-request usage as a capability

Sixteenth in the capability-conversion series (#828-#836, #838-#842, #844). First claim of the after_model_request seam.

The feature

invoke_agent_with_model reports token usage per model request (per_request_usage -- "use per_request_usage for pricing") and the final request's context size (final_context_tokens). On main, both are derived after the run by walking result.new_messages() for ModelResponse objects (code_puppy/tools/subagent_usage_metrics.py).

The conversion

New code_puppy/agents/_subagent_usage.py:

  • PerRequestUsageCapture -- one instance per invocation. for_run hands each agent.run() a fresh run-scoped collector (proven necessary: responses accumulate across runs on a shared instance, and streaming-retry re-entry means multiple run() calls per invocation). The collector records each ModelResponse on after_model_request and stows (result, responses) into a one-slot capture at after_run -- which only fires for runs that produced a result, so failed attempts never pollute the slot.
  • consume(result) is read-and-clear and double-gated:
    • Identity gate (captured result is result, feat(agents): promote run-end telemetry to a RunTelemetry capability (after_run seam) #844 precedent): rejects stale captures and guest wrappers that bypass capabilities.
    • Consistency gate: rejects captures whose response sequence no longer identity-matches the result's recorded ModelResponses. This is not paranoia -- mid-run history rewrites are real: a ProcessHistory pass that drops one of this run's own responses (the shape mid-run compaction produces on a long sub-agent run) leaves the seam having seen N responses while new_messages() records N-1. The eager walk on main never reported the dropped response; without the gate the capability would report a phantom usage entry. Pinned by test_consistency_gate_rejects_mid_run_history_rewrite, which reproduces the divergence empirically.
  • Every rejection converges on the call site's eager fallback -- the literal pre-capability new_messages() walk -- and both paths feed the same extraction helpers (extract_per_request_usage / extract_final_context_tokens, untouched), so both paths are byte-identical.

Seam mechanics (2.31.0, verified empirically)

  • after_model_request fires once per model response, for both streamed and non-streamed requests (all three _finish_handling call sites), and receives the identical ModelResponse object that _append_response commits to run state.
  • CombinedCapability applies after-hooks in reverse list order (onion semantics) -- caught by the reviewer in pass 1. The capture therefore goes first in capabilities=[...]: it executes last in after_model_request and records the response object that actually reaches run state even if another capability replaced it. Pinned both ways: the production ordering owns replacements; the adversarial misordering is disowned by the consistency gate and falls back rather than reporting pre-replacement objects.
  • for_run is async in 2.31.0 and resolves once per run(); after_run fires on the resolved run capability with the identical AgentRunResult the caller receives (re-confirmed from feat(agents): promote run-end telemetry to a RunTelemetry capability (after_run seam) #844).

Scope & bounded non-moves

  • Run-total usage_metrics (result.usage) and duration_ms stay at the call site. The timing deliberately spans streaming-retry attempts -- multiple agent.run() calls -- so it cannot live inside a per-run capability with parity.
  • The capability is only constructed when include_usage_metrics is set (invoke_agent_with_model only); plain invoke_agent invocations get an empty splice and zero new machinery -- mirroring the build_tool_output_limits conditional-splice pattern.
  • Main-path telemetry untouched (that is feat(agents): promote run-end telemetry to a RunTelemetry capability (after_run seam) #844's territory, in reverse).
  • Extraction helpers in subagent_usage_metrics.py untouched -- all 42 existing helper tests pass unchanged.

Behavioral divergences

None for real AgentRunResults (whose new_messages() is a stable view over run state). The observation moved to the seam; the reported values are computed by the same helpers over the same objects, and every case where the seam data could differ from the eager walk is gated back onto the eager walk. The consistency gate adds one new_messages() call, observable only by a pathological result object with stateful new_messages() semantics.

Tests

18 contract tests in tests/agents/test_per_request_usage_capability.py: seam capture identity/order (streamed + non-streamed), read-and-clear, identity gate, consistency gate (incl. the mid-run rewrite reproduction and a new_messages()-raising result preserving the old failure mode), per-run isolation across sequential runs, extraction parity vs the eager walk (non-vacuous: FunctionModel reports real usage), construction helper, spec-constructibility, and a call-site wiring pin.

Merge-order note

Same story as the rest of the series: this PR touches the sub-agent capabilities=[...] block that #828-#842 also graze. Whoever lands last eats a trivial rebase. This one additionally shares the success-boundary region of _invoke_agent_impl with #842 (session persistence) -- also trivial.

…equest capability

invoke_agent_with_model's per_request_usage and final_context_tokens were
derived after the run by walking result.new_messages() for ModelResponse
objects. Move the observation onto pydantic-ai's after_model_request
capability seam: PerRequestUsageCapture records each response at the
exact moment the run appends it to state (verified identical object for
streamed and non-streamed requests on 2.31.0), and hands the invocation
layer an owned capture at the run boundary.

consume() is read-and-clear and double-gated: an identity gate rejects
stale captures (streaming-retry re-entry, guest wrappers bypassing
capabilities) and a consistency gate rejects captures whose response
sequence no longer matches the result's recorded responses -- mid-run
compaction can summarize away a long run's own earlier responses, and
the eager walk never saw those. Every rejection falls back to the
pre-capability new_messages() walk through the same extraction helpers,
so both paths are byte-identical.

Scope: run-total usage_metrics and duration_ms stay at the call site --
the timing spans streaming-retry attempts (multiple agent.run() calls)
and cannot live inside a per-run capability with parity. The capability
is only constructed when include_usage_metrics is set, mirroring the
build_tool_output_limits conditional-splice pattern.
… order)

Review pass 1 finding: CombinedCapability.after_model_request iterates
reversed(self.capabilities), so last-in-list executes FIRST -- the
previous comment had the onion inside out. Move the capture to first
position so it executes last and records the response object that
actually reaches run state even if another capability replaces it.

Pin the semantics both ways: a replacer executing before the capture is
observed and owned; the adversarial misordering is disowned by the
consistency gate and falls back to the eager walk instead of reporting
pre-replacement objects.

Also: narrow the byte-identical claim to real AgentRunResult semantics
(the consistency gate adds one new_messages() call that only a
pathological stateful result object could observe), and modernize the
typing spellings (list/tuple/X | None).
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant