fix: zero-fill known status codes absent from a returned minute

[shreekarshetty]

Summary

Root-caused the CloudflareZone5xxZscoreWarn noise to idle status-code series silently dropping out of aggregate queries. Zero-fills them using Cloudflare's own response as the source of truth, instead of guessing with a timer.

Claude details

Root cause

Cloudflare's Adaptive Groups API only returns a (zone, status) row in a minute when that status had at least one request. Engine.Ingest only re-emits a counter when an Observation for that exact (key, bucket) arrives, so a rare 5xx code (501, 507, 520, 523, 525, 555 on supabase.co; 504, 530 on snapcloud.dev) that goes quiet for a few minutes just stops getting pushed. Once the gap outlives VictoriaMetrics' staleness window, the series drops out of sum by (zone) (...) entirely, making the zone's total 5xx count appear to fall - then jump back the instant the code reappears. That swing is what trips cloudflare:zone_5xx_zscore.

Verified against the three real alerts (08:45, 09:15, 09:25 UTC, 2026-08-14):

• supabase.co: status 555 gap 09:14:30-09:32 UTC (17.5min), 507 gap 09:24-09:36 UTC (12min), plus shorter gaps on 501/520/525
• snapcloud.dev: status 504 gaps of 12-14min, 530 appears for a single 30s sample then vanishes

Fix

A superseded earlier version of this fix (#47, closed) used a timer: re-push every known key's last value every tick unless it's been idle past a cap, to bound (but not eliminate) the risk of masking a genuine sustained outage. That was solving with a guess something Cloudflare already answers directly: a minute row that comes back at all is a complete, authoritative list of every status that occurred - a known status missing from it is a confirmed zero, not an ambiguous gap.

flattenHTTPAdaptiveGroups now tracks, per zone, every status code ever seen and zero-fills any that a given minute's response doesn't mention, feeding a real Value: 0 Observation through the existing (already-tested) counter-chain/tracker pipeline - no new Engine API surface. Because zero-fill only ever fires for a (zone, minute) a fetch actually, successfully returned data for, it can't mask a real outage the way the timer could: if a chunk fetch fails outright, this never runs for that zone, and its keys fall through to real staleness exactly as today - CloudflareZoneRequestsMetricMissing still catches a genuine sustained failure at its normal ~10 minute speed, no grace period needed.

Testing

cfetch/zerofill_test.go - TDD: tests written first (confirmed failing to compile without the knownStatuses field / method), then implemented. Covers: zero-fill of a previously-seen-now-absent status, no zero-fill before a status has ever been seen, zero-fill knowledge isolated per zone, and the existing enabled metric filter still works. Full suite (go test ./...), go vet ./..., and gofmt all clean.

Scope

Left cfgql.FetchZones's silent per-chunk error swallowing alone (real defect - no error propagation, no metric - but this fix's safety doesn't depend on it, since zero-fill inherently can't fire off a failed fetch). Also left flattenHTTP1mGroups's equivalent non-v2 cloudflare_zone_requests_status breakdown untouched - same latent issue, but nothing currently alerts on that metric, so it's out of scope here.

⚡ Built with Claude Code