Bill eval-driven generation as evaluation spend

[snopoke]

Closes #3985.

Product Description

Cost reporting now includes what an evaluation run's bot generation spent. Team totals rise for teams running generation evals — on money that was previously not recorded at all. No chatbot, participant or conversation gets more expensive.

Technical Description

The issue assumed generation was already recorded as source=chat and that this PR would reclassify it. It wasn't recorded at all: EvaluationChannel has run with no tracers since Oct 2025, so generation wrote no Trace and therefore no UsageRecord. The fix is a write-side addition, not a reclassification — hence no backfill.

Turning OCSTracer back on for eval channels would resurrect what was deliberately switched off (one Trace per evaluated message, outliving the pruned eval session, plus error notifications on eval failures). Instead there's a UsageOnlyTracer — the billing half of OCSTracer with the rest dropped — installed as the eval channel's sole tracer. ADR-0050 has the full reasoning and rejected alternatives.

Worth review attention:

• _usage_source() on OCSTracer classifies by session.platform, which the issue settled as the right signal. It's belt-and-braces: eval generation never reaches this tracer, but other paths reach an eval session (a static trigger firing on one). Read at finalisation, since set_session back-fills mid-trace.
• usage_tracer: Tracer | None on handle_evaluation_message is deliberately the general type rather than UsageOnlyTracer, to keep apps/channels off cost tracking. It means nothing stops a caller passing an OCSTracer and reintroducing the trace rows this avoids.
• _create_trace_service on EvaluationChannel sets self._usage_tracer before super().__init__(), which builds the service from it.

An eval session's cost page now reports $0 (both halves are source=EVALUATION, session_usage is chat-only) — #3981. The admin report's token and cost halves diverge further until #3984 moves token reporting onto UsageRecord.

Migrations

N/A — no schema change.

Docs and Changelog

• This PR requires docs/changelog update

Changelog note: per-team cost totals now include evaluation bot generation, which was previously unrecorded. Existing rows are unchanged.

[codecov-commenter]

❌ 1 Tests Failed:

Tests completed	Failed	Passed	Skipped
5204	1	5203	2

View the top 1 failed test(s) by shortest run time

apps/pipelines/tests/test_add_start_end_nodes.py::test_remove_start_end_nodes

Stack Traces | 0.068s run time

.../pipelines/tests/test_add_start_end_nodes.py:259: in test_remove_start_end_nodes
    remove_all_start_end_nodes(Node)
.../migrations/utils/migrate_start_end_nodes.py:19: in remove_all_start_end_nodes
    _set_new_nodes(pipeline, Node)
.../migrations/utils/migrate_start_end_nodes.py:129: in _set_new_nodes
    "type": node.data.type,
            ^^^^^^^^^^^^^^
E   AttributeError: 'NoneType' object has no attribute 'type'

To view more test analytics, go to the Test Analytics Dashboard
_{📋 Got 3 mins? Take this short survey to help us improve Test Analytics.}

[coderabbitai]

📝 Walkthrough

Walkthrough

Evaluation-driven bot generation now receives an evaluation-specific usage tracer that records UsageRecord rows as evaluation spend without creating Trace rows. Existing traced usage is classified as chat or evaluation based on the final session platform. Tests cover tracer wiring, source classification, attribution, trace absence, and reporting totals. Documentation updates describe the reconciliation between trace-based token reporting and usage-record cost reporting.

Estimated code review effort: 3 (Moderate) | ~25 minutes

Possibly related issues

• Consolidate token reporting on UsageRecord instead of Trace #3984 — Clarifies the token-reporting reconciliation behavior updated in this PR.

Possibly related PRs

• dimagi/open-chat-studio#3978 — Adds the related evaluation-session token reporting changes documented here.
• dimagi/open-chat-studio#3573 — Provides the cost-tracking foundation used by UsageOnlyTracer.
• dimagi/open-chat-studio#3473 — Introduces related trace-based token aggregation behavior in apps/admin/queries.py.

Suggested reviewers: smittiec, codescene-delta-analysis

🚥 Pre-merge checks | ✅ 4 | ❌ 1

❌ Failed checks (1 warning)

Check name	Status	Explanation	Resolution
Docstring Coverage	⚠️ Warning	Docstring coverage is 66.67% which is insufficient. The required threshold is 80.00%.	Write docstrings for the functions missing them to satisfy the coverage threshold.

✅ Passed checks (4 passed)

Check name	Status	Explanation
Linked Issues check	✅ Passed	The code matches #3985 by billing eval-driven generation as evaluation spend, using eval platform at finalization and avoiding backfills.
Out of Scope Changes check	✅ Passed	The changes stay focused on eval spend attribution, tests, docs, and ADR updates, with no obvious unrelated scope creep.
Title check	✅ Passed	The title is concise and accurately summarizes the main change: billing eval-driven generation as evaluation spend.
Description check	✅ Passed	The description covers Product, Technical, Migrations, and Docs/Changelog sections and provides the key implementation details.

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands.}

[coderabbitai]

Actionable comments posted: 1

🧹 Nitpick comments (1)

apps/evaluations/tests/test_generation_usage.py (1)

14-14: 📐 Maintainability & Code Quality | 🔵 Trivial | ⚡ Quick win

Lazy-load the LangChain test dependency.

langchain_core matches the prohibited module-level langchain_* import pattern and is only needed by _generation_service.

Proposed fix

-from langchain_core.messages import AIMessage
-
 def _generation_service(input_tokens: int = 1000, output_tokens: int = 500):
+    from langchain_core.messages import AIMessage
+

As per coding guidelines, local imports are allowed to “reduce startup time”; as per path instructions, heavy langchain_* imports must be lazy-loaded.

🤖 Prompt for AI Agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@apps/evaluations/tests/test_generation_usage.py` at line 14, Remove the
module-level AIMessage import and lazy-load AIMessage inside
_generation_service, where it is used. Keep the existing message construction
behavior unchanged while ensuring no langchain_* dependency is imported during
module initialization.

Sources: Coding guidelines, Path instructions

🤖 Prompt for all review comments with AI agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@docs/adr/0049-eval-driven-generation-is-evaluation-spend.md`:
- Around line 1-5: Add YAML frontmatter to ADR-0049 with status: active while
the ADR is proposed and under development; update it to stable after shipping
and before extraction, preserving the existing HTML status marker.

---

Nitpick comments:
In `@apps/evaluations/tests/test_generation_usage.py`:
- Line 14: Remove the module-level AIMessage import and lazy-load AIMessage
inside _generation_service, where it is used. Keep the existing message
construction behavior unchanged while ensuring no langchain_* dependency is
imported during module initialization.

🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

• Push a commit to this branch (recommended)
• Create a new PR with the fixes

---

ℹ️ Review info

⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Pro Plus

Run ID: 9908b496-5e3b-48e4-9e80-5b9ed0ac1b16

📥 Commits

Reviewing files that changed from the base of the PR and between 62bc5b2 and dbdda48.

📒 Files selected for processing (18)

• apps/admin/queries.py
• apps/channels/evaluation_channel.py
• apps/channels/tasks.py
• apps/channels/tests/channels/concrete/test_evaluation_channel.py
• apps/cost_tracking/models.py
• apps/cost_tracking/services/recorder.py
• apps/cost_tracking/services/reporting.py
• apps/evaluations/tasks.py
• apps/evaluations/tests/test_evaluation_tasks.py
• apps/evaluations/tests/test_generation_usage.py
• apps/evaluations/usage.py
• apps/service_providers/tracing/const.py
• apps/service_providers/tracing/ocs_tracer.py
• apps/service_providers/tracing/tests/test_ocs_tracer_cost.py
• apps/service_providers/tracing/usage_tracer.py
• docs/adr/0049-eval-driven-generation-is-evaluation-spend.md
• docs/adr/index.md
• mkdocs.yml

[claude]

Code review

No issues found. Checked for bugs and CLAUDE.md compliance.

[SmittieC]

Cool