feat(hybridcloud): Tag delivery outcomes and latency with their dispatcher and mode

[vaind]

Tags deliver_webhooks.delivery and deliver_webhooks.delivery_time_ms with the dispatcher and dispatch regime of the drain that produced each one.

Why

The claim_dispatch_rollout selects whole integrations by hash (md5(provider:integration_id)), so at a low percentage its effect on any global total is far smaller than that total's own variance. Measured over 2h:

	value
races	321,057 / 2h
per-bucket mean / sd	1,338 / 808 (60%)
effect of a 1% cohort	~13 per bucket

That is roughly 60x below one standard deviation. Reading the rollout from global totals needs ~12% of traffic over a 2h window, or ~3.4% over a day, before the signal clears noise — and that estimate is optimistic, since it treats buckets as independent when traffic is autocorrelated.

Segmenting by cohort removes the constraint. Comparing the claim cohort's rates against the lease cohort's is a rate comparison rather than a shift in a sum, so it stays valid at any rollout percentage, including 1%.

delivery was tagged outcome and provider only, and delivery_time_ms region_sent_to, provider and github_event_and_action. Once a drain task starts it has no idea which dispatcher enqueued it or under which regime, so neither segmentation was possible.

What

Both drain tasks accept dispatcher and mode, and all four dispatch sites (push/scheduler × claim/lease) forward them. _dispatch_tags builds the pair once per drain and every delivery outcome carries it — 15 emit sites, including the ones in deliver_message, _discard_if_stale, and the parallel result handler.

_record_delivery_time_metrics carries the same pair. Latency is the quantity the claim regime is meant to move, and it was readable only in aggregate: a global p50 cannot show a cohort effect while the cohort is a small share of traffic. It is measured from date_added, so it spans queue wait, retry backoff and the request — the span dispatch acts on. Worth noting delivery_time_ms is routed to the precise backend and is a native Datadog distribution rather than the histogram the other metrics.distribution calls in this file produce, so percentiles are genuinely queryable per tag combination; the sibling send_request timer has no median at all.

A DispatchMode StrEnum replaces the bare "claim"/"lease" literals now that the value is a task argument rather than a local metric tag.

This also supersedes the derived push-vs-scheduler item split currently on the dashboard, which infers push volume by subtracting scheduler claims from the delivery total — an estimate that over-credits the scheduler whenever a claimed batch is not fully drained.

Deploy notes

Both metrics are in allowlist mode (exclude_tags_mode: false), so dispatcher and mode are dropped at intake until they are added:

metric	tag config	series (24h)	during ramp	at 100%
delivery	✅ dispatcher, mode added	48	157	94
delivery_time_ms	❌ still github_event_and_action, provider, region_sent_to, sentry_region	158	543	313

Projected from the fan-out dispatch already exhibits — it carries both tags today, so applying its dispatcher/mode volume shares to each series' own delivery volume gives 3.44x during the ramp, against a 4x ceiling the low-volume tail never reaches. Rare outcomes (max_age, attempts_exceed) put the true delivery base somewhat above a 24h sample.

The peak is transient. Both modes are live only while the rollout is partial; at 100% every mailbox is mode:claim, the tag collapses to a single value, and the count falls back to the dispatcher split alone.

delivery_time_ms is the more expensive of the two — it is a percentile-enabled distribution and already carries the high-cardinality github_event_and_action. Listing dispatcher alone holds it to 313 and still separates push, which fires on arrival, from the scheduler's 10s cycle — at the cost of the rollout A/B. Both are listed here on the assumption that the structural difference is worth measuring alongside the rollout; dropping one is a config change, not a code change.

Deploy order does not matter. Both metrics keep working unchanged while the tags are unlisted, and queries that do not group by them are unaffected. Adding the tags before this deploys is a no-op since nothing emits them yet; adding them after costs only the window in between, which simply has no cohort dimension. The tags only need to be in place before the rollout ramps.

Tasks already queued when this deploys arrive without the arguments and tag unknown rather than omitting the keys — a tag absent from some series fails a group_by for the whole query rather than showing a gap.

New task kwargs reach old workers during the rolling window, as with #121221.

Scope

No behavior change: no query, delivery decision, or claim bound is altered. Tests cover attribution on the sequential and parallel drains for both metrics, the race path (the rollout's primary benefit signal), and the unknown fallback; the four dispatch tests now also assert what they forward.

Part of CW-1830. Follow-up to #121882.

[linear-code]

CW-1830