Skip to content

feat(models): add Claude Opus 5 + Sonnet 5 and fix silent $0 cost budget bypass - #1531

Merged
aaight merged 1 commit into
devfrom
feat/mng-1770-opus5-sonnet5-pricing
Aug 4, 2026
Merged

feat(models): add Claude Opus 5 + Sonnet 5 and fix silent $0 cost budget bypass#1531
aaight merged 1 commit into
devfrom
feat/mng-1770-opus5-sonnet5-pricing

Conversation

@aaight

@aaight aaight commented Aug 4, 2026

Copy link
Copy Markdown
Collaborator

Summary

Adds claude-opus-5 + claude-sonnet-5 across all three hand-maintained model tables and fixes the root-cause silent $0 cost fallback that let unpriced models bypass workItemBudget enforcement.

Implements the plan in MNG-1770.

What changed

New models (all 3 drifting tables, keyed differently — which is why they drift)

  • Dropdown / allowlist (src/backends/claude-code/models.ts): added claude-opus-5 (above the Opus 4.x block) and claude-sonnet-5 (above the Sonnet 4.x block). No [1m] variants — both default to 1M context, mirroring claude-fable-5. CLAUDE_CODE_MODEL_IDS derives automatically, so resolveClaudeModel() accepts them with no allowlist edit.
  • Pricing (src/utils/llmMetrics.ts): anthropic:claude-opus-5 = {5, 25, 0.5}, anthropic:claude-sonnet-5 = {3, 15, 0.3} (standard post-intro rates — intro $2/$10 runs through 2026-08-31; over-reporting is the budget-safe direction).
  • Rate limits (src/config/rateLimits.ts): each gets its own row (Opus 5: 50 RPM / 10K TPM; Sonnet 5: 50 RPM / 40K TPM) — prefix matching would never reach an Opus/Sonnet 4.x row.

Backfilled two pre-existing pricing gaps (both were running unbudgeted at $0)

  • Bare anthropic:claude-opus-4-6 (only the [1m] variant was priced).
  • Bare anthropic:claude-haiku-4-5 (claude-haiku-4-5-20251001 is in the dropdown but only claude-haiku-3-5 was priced). Seeded {1, 5, 0.1}.

Root-cause fix: make the $0 pricing miss loud (src/utils/llmMetrics.ts)

calculateCost previously failed open (if (!pricing) return 0), silently disabling workItemBudget (checkBudgetExceeded never trips on $0). Now a missing row emits a logger.warn + captureException (tag model_pricing_missing, level: 'warning', extra: { model }), deduped via a module-level Set so it fires once per unique model per process (calculateCost runs per turn; workers are ephemeral). Deliberately non-fatal — it's a hot-path pure utility called across all three engines, so throwing would crash runs for any not-yet-priced model. A legitimate {input: 0, output: 0} row (e.g. huggingface:*) is truthy and stays silent.

Drift-guard test + toPricingKey export

  • Exported toPricingKey from messageProcessing.ts and MODEL_PRICING from llmMetrics.ts.
  • New drift-guard test iterates every CLAUDE_CODE_MODEL_IDS entry through the exact runtime toPricingKey transform and asserts a MODEL_PRICING row exists — this single test would have caught both pre-existing gaps and will catch the next one.

Default model bump (reviewer decision — Step 6)

  • DEFAULT_CLAUDE_CODE_MODEL: claude-sonnet-4-5-20250929claude-sonnet-5; catalog label Default (Sonnet 4.5)Default (Sonnet 5). Blast radius is bounded to the claude-code engine for override-less projects. Isolated to 2 lines across models.ts + catalog.ts so it can be reverted independently if a conservative rollout is preferred.

Testing

  • llmMetrics.test.ts: Opus 5 / Sonnet 5 pricing math, backfilled-row assertions, and the loud-miss path (warn + Sentry fired once, deduped on a second call, silent for zero-priced rows).
  • rateLimits.test.ts: exact-match rows for Opus 5 / Sonnet 5 (existing field/margin loops cover them too).
  • claude-code.test.ts: model-count + new-ID toContain checks, resolveClaudeModel passthrough, and the new MODEL_PRICING coverage drift guard.
  • Full unit suite green (10548 passed). The only failures on this worker were pre-existing web/-component tests failing because web/node_modules wasn't installed by root npm install — all pass after cd web && npm install, and none touch changed files (filed as friction MNG-1771). npm run typecheck + Biome clean on changed files.

Notes / risks

  • Loud-miss now fires across all engines (claude-code, codex, LLMist). Intended — surfaces gaps. captureException is a no-op when SENTRY_DSN is unset, so CI stays quiet. Verified no existing test asserts the absence of a warn/Sentry during a cost calc.
  • Unverified rates (Haiku 4.5 $1/$5, Sonnet 5 standard vs intro, exact 1M-context routing IDs) are proposed values — a mis-rate is a mis-report; a missing row is a budget bypass, which the drift test now guards.
  • Single-source-of-truth catalog refactor (deriving all three tables from one keyed source) is intentionally out of scope; the drift-guard mitigates the immediate risk — recommend a follow-up issue.

🤖 Generated with Claude Code

🕵️ claude-code · claude-opus-4-8 · run details

@codecov

codecov Bot commented Aug 4, 2026

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.

📢 Thoughts on this report? Let us know!

@nhopeatall nhopeatall left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Summary

APPROVE — verified correct. Adds Opus 5 / Sonnet 5 across all three drifting tables, backfills two real pre-existing $0 gaps (bare claude-opus-4-6, bare claude-haiku-4-5), and fixes the root-cause silent-$0 budget bypass with a loud-but-non-fatal miss + a drift-guard test. Matches the MNG-1770 plan step-for-step.

What I verified

  • Drift guard is complete. All 13 CLAUDE_CODE_MODEL_IDS map through the exact runtime toPricingKey transform to an existing MODEL_PRICING row — including the two backfills and the date-stripped claude-sonnet-4-5-20250929anthropic:claude-sonnet-4-5 and claude-haiku-4-5-20251001anthropic:claude-haiku-4-5. The test imports the exported runtime toPricingKey rather than re-implementing the regex, so it can't drift from runtime.
  • Loud-miss is genuinely non-fatal. captureException early-returns when SENTRY_DSN is unset (src/sentry.ts:13) and the Sentry SDK swallows its own errors, so an unpriced model now warns + captures once per process and still returns 0 — it cannot crash the hot path shared by all three engines. The dedup Set keying is correct, and the truthy {input:0,output:0} huggingface row stays silent as intended.
  • Rate-limit rows are consistent with family conventions (Opus 0.85 / 10K TPM, Sonnet 0.9 / 40K TPM) and correctly get their own rows since getRateLimitForModel prefix matching can't reach an Opus/Sonnet 4.x row.
  • Sentry tag { source: 'model_pricing_missing' } matches the established { source: <snake_case> } convention used throughout src/.
  • Default-model bump is isolated to 2 lines; DEFAULT_CLAUDE_CODE_MODEL is a known model ID passes because claude-sonnet-5 is in the allowlist, its pricing + rate-limit rows exist, and no test asserts the old Default (Sonnet 4.5) label. Other claude-sonnet-4-5-20250929 references are independent fixtures, not the default constant.
  • CI is 7/7 green; llmMetrics.test.ts correctly switched from describe.concurrentdescribe so the shared-mock loud-miss assertions stay deterministic.

Notes (no action required — all documented in the PR)

  • Default model availability: the only real blast radius is that override-less claude-code projects begin using claude-sonnet-5 immediately. Worth confirming it's a live routing ID at the Anthropic API before merge — a not-yet-available model would surface as run-time API errors (the allowlist passthrough won't catch it). Isolated for easy revert if a conservative rollout is preferred.
  • Sonnet 5 intro pricing: seeding standard $3/$15 over-reports vs the $2/$10 intro rate through 2026-08-31 (~4 weeks out). This is the budget-safe direction (never under-reports) and is documented — fine to leave.
  • Codex/LLMist coverage: the loud-miss now also fires for any unpriced model on those paths, which the drift guard (claude-code-allowlist-specific) doesn't cover. That's the intended gap-surfacing net, deduped and a no-op without a DSN.

🕵️ claude-code · claude-opus-4-8 · run details

@aaight
aaight merged commit f65a5fb into dev Aug 4, 2026
9 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants