feat(hybridcloud): Split backlog metrics by dispatch regime and report depth quantiles

[vaind]

Tags every webhookpayload.mailbox.* metric with the dispatch regime that would drain the mailbox, and adds a depth distribution alongside the existing max.

Why

While claim_dispatch_rollout ramps, the question the backlog charts are asked is "is the claim cohort's backlog burning down while lease's stays flat?" They cannot answer it. A provider-only rollup averages the two regimes into a single line that describes neither, so the burn-down has to be inferred from a global total whose own variance dwarfs the cohort's contribution — the same problem #121982 fixed for delivery outcomes and latency.

This came out of reading the 50% rollout. The remaining backlog concentrated in one event type, and there was no way to tell whether that was a cohort that had not been treated yet or simply the highest-volume event type draining slowest. pending_count carries no regime, so the chart cannot distinguish them.

max_depth has a related blind spot. It reports the same number whether a single mailbox is stuck at 300 or the provider's entire backlog has shifted deeper, and those want different responses — one is a mailbox to go look at, the other is a capacity problem.

What

mode (claim / lease) on pending_count, active_count, max_depth and oldest_pending_age_seconds, and a new mailbox.depth_quantile gauge tagged quantile:p50|p90|p99.

The regime comes from use_claim_dispatch rather than a restatement of the hash, so the cohort this reports can never drift from the cohort that actually claims. That is why the predicate stops being private — this is the second legitimate caller, and a copied md5 would silently desync the metric from real dispatch the moment the bucketing key changed.

Quantiles are nearest-rank, so every value returned is a depth some mailbox actually has; an interpolated 287.4 names none of them.

Cost

Deliberately small, because the expensive part is already paid. record_mailbox_depth_metrics already aggregates every mailbox under a statement timeout and already walks the per-mailbox depths in memory. The regime tag adds an md5 per row and the quantiles add one sort per (provider, mode) — against a full-table aggregate running every 5 minutes.

Series cost is 2x on the four existing mailbox metrics, plus 3 per (provider, mode) for the quantiles. Reporting quantiles rather than emitting a distribution is what keeps that bounded: a metrics.distribution per mailbox would put a series count proportional to the backlog behind the metric that exists to measure the backlog.

Caveat

The regime is derived per run, not stored, so a rollout change re-attributes existing backlog on the next pass. The tag answers "which regime owns this backlog now", not "which one produced it" — the burn-down curve steps at each rollout change rather than tracking a fixed population. That is the right semantics for the question being asked of it, but it is worth knowing before reading a discontinuity as a regression.

Scope

No behavior change: no query, delivery decision, or claim bound is altered, and the aggregate query is untouched. use_claim_dispatch is a pure rename, no call-site semantics change.

Part of CW-1830. Follow-up to #121982.

[linear-code]

CW-1830