feat(models): add Claude Opus 5 + Sonnet 5 and fix silent $0 cost budget bypass - #1531
Merged
Conversation
Codecov Report✅ All modified and coverable lines are covered by tests. 📢 Thoughts on this report? Let us know! |
nhopeatall
approved these changes
Aug 4, 2026
nhopeatall
left a comment
Collaborator
There was a problem hiding this comment.
Summary
APPROVE — verified correct. Adds Opus 5 / Sonnet 5 across all three drifting tables, backfills two real pre-existing $0 gaps (bare claude-opus-4-6, bare claude-haiku-4-5), and fixes the root-cause silent-$0 budget bypass with a loud-but-non-fatal miss + a drift-guard test. Matches the MNG-1770 plan step-for-step.
What I verified
- Drift guard is complete. All 13
CLAUDE_CODE_MODEL_IDSmap through the exact runtimetoPricingKeytransform to an existingMODEL_PRICINGrow — including the two backfills and the date-strippedclaude-sonnet-4-5-20250929→anthropic:claude-sonnet-4-5andclaude-haiku-4-5-20251001→anthropic:claude-haiku-4-5. The test imports the exported runtimetoPricingKeyrather than re-implementing the regex, so it can't drift from runtime. - Loud-miss is genuinely non-fatal.
captureExceptionearly-returns whenSENTRY_DSNis unset (src/sentry.ts:13) and the Sentry SDK swallows its own errors, so an unpriced model now warns + captures once per process and still returns 0 — it cannot crash the hot path shared by all three engines. The dedupSetkeying is correct, and the truthy{input:0,output:0}huggingface row stays silent as intended. - Rate-limit rows are consistent with family conventions (Opus 0.85 / 10K TPM, Sonnet 0.9 / 40K TPM) and correctly get their own rows since
getRateLimitForModelprefix matching can't reach an Opus/Sonnet 4.x row. - Sentry tag
{ source: 'model_pricing_missing' }matches the established{ source: <snake_case> }convention used throughoutsrc/. - Default-model bump is isolated to 2 lines;
DEFAULT_CLAUDE_CODE_MODEL is a known model IDpasses becauseclaude-sonnet-5is in the allowlist, its pricing + rate-limit rows exist, and no test asserts the oldDefault (Sonnet 4.5)label. Otherclaude-sonnet-4-5-20250929references are independent fixtures, not the default constant. - CI is 7/7 green;
llmMetrics.test.tscorrectly switched fromdescribe.concurrent→describeso the shared-mock loud-miss assertions stay deterministic.
Notes (no action required — all documented in the PR)
- Default model availability: the only real blast radius is that override-less claude-code projects begin using
claude-sonnet-5immediately. Worth confirming it's a live routing ID at the Anthropic API before merge — a not-yet-available model would surface as run-time API errors (the allowlist passthrough won't catch it). Isolated for easy revert if a conservative rollout is preferred. - Sonnet 5 intro pricing: seeding standard $3/$15 over-reports vs the $2/$10 intro rate through 2026-08-31 (~4 weeks out). This is the budget-safe direction (never under-reports) and is documented — fine to leave.
- Codex/LLMist coverage: the loud-miss now also fires for any unpriced model on those paths, which the drift guard (claude-code-allowlist-specific) doesn't cover. That's the intended gap-surfacing net, deduped and a no-op without a DSN.
🕵️ claude-code · claude-opus-4-8 · run details
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Adds
claude-opus-5+claude-sonnet-5across all three hand-maintained model tables and fixes the root-cause silent$0cost fallback that let unpriced models bypassworkItemBudgetenforcement.Implements the plan in MNG-1770.
What changed
New models (all 3 drifting tables, keyed differently — which is why they drift)
src/backends/claude-code/models.ts): addedclaude-opus-5(above the Opus 4.x block) andclaude-sonnet-5(above the Sonnet 4.x block). No[1m]variants — both default to 1M context, mirroringclaude-fable-5.CLAUDE_CODE_MODEL_IDSderives automatically, soresolveClaudeModel()accepts them with no allowlist edit.src/utils/llmMetrics.ts):anthropic:claude-opus-5={5, 25, 0.5},anthropic:claude-sonnet-5={3, 15, 0.3}(standard post-intro rates — intro $2/$10 runs through 2026-08-31; over-reporting is the budget-safe direction).src/config/rateLimits.ts): each gets its own row (Opus 5: 50 RPM / 10K TPM; Sonnet 5: 50 RPM / 40K TPM) — prefix matching would never reach an Opus/Sonnet 4.x row.Backfilled two pre-existing pricing gaps (both were running unbudgeted at $0)
anthropic:claude-opus-4-6(only the[1m]variant was priced).anthropic:claude-haiku-4-5(claude-haiku-4-5-20251001is in the dropdown but onlyclaude-haiku-3-5was priced). Seeded{1, 5, 0.1}.Root-cause fix: make the $0 pricing miss loud (
src/utils/llmMetrics.ts)calculateCostpreviously failed open (if (!pricing) return 0), silently disablingworkItemBudget(checkBudgetExceedednever trips on $0). Now a missing row emits alogger.warn+captureException(tagmodel_pricing_missing,level: 'warning',extra: { model }), deduped via a module-levelSetso it fires once per unique model per process (calculateCost runs per turn; workers are ephemeral). Deliberately non-fatal — it's a hot-path pure utility called across all three engines, so throwing would crash runs for any not-yet-priced model. A legitimate{input: 0, output: 0}row (e.g.huggingface:*) is truthy and stays silent.Drift-guard test +
toPricingKeyexporttoPricingKeyfrommessageProcessing.tsandMODEL_PRICINGfromllmMetrics.ts.CLAUDE_CODE_MODEL_IDSentry through the exact runtimetoPricingKeytransform and asserts aMODEL_PRICINGrow exists — this single test would have caught both pre-existing gaps and will catch the next one.Default model bump (reviewer decision — Step 6)
DEFAULT_CLAUDE_CODE_MODEL:claude-sonnet-4-5-20250929→claude-sonnet-5; catalog labelDefault (Sonnet 4.5)→Default (Sonnet 5). Blast radius is bounded to theclaude-codeengine for override-less projects. Isolated to 2 lines acrossmodels.ts+catalog.tsso it can be reverted independently if a conservative rollout is preferred.Testing
llmMetrics.test.ts: Opus 5 / Sonnet 5 pricing math, backfilled-row assertions, and the loud-miss path (warn + Sentry fired once, deduped on a second call, silent for zero-priced rows).rateLimits.test.ts: exact-match rows for Opus 5 / Sonnet 5 (existing field/margin loops cover them too).claude-code.test.ts: model-count + new-IDtoContainchecks,resolveClaudeModelpassthrough, and the newMODEL_PRICINGcoverage drift guard.10548 passed). The only failures on this worker were pre-existingweb/-component tests failing becauseweb/node_moduleswasn't installed by rootnpm install— all pass aftercd web && npm install, and none touch changed files (filed as friction MNG-1771).npm run typecheck+ Biome clean on changed files.Notes / risks
captureExceptionis a no-op whenSENTRY_DSNis unset, so CI stays quiet. Verified no existing test asserts the absence of a warn/Sentry during a cost calc.🤖 Generated with Claude Code
🕵️ claude-code · claude-opus-4-8 · run details