You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Measured magnitudes (orphan-swept cells, W=128, 4CPU-capped postgres): at 8,000/s offered (past the ~6.5k drain knee), v0.6.0 drains 7,199/s (backlog →231k, p99 ~18.6s) and the 0.7 ledger branch drains 7,641/s (backlog →115k, p99 ~7.9s). Drain stays near the healthy ceiling; the dominant overload symptom is unbounded backlog + multi-second latency (the #341 backpressure gap). The single-generation yield ceiling below is real (deterministic seeder, identical on both versions) but requires deeper fragmentation than these shapes produce — it is a latent ceiling that mainly slows recovery from extreme backlogs.
Found while gating the 0.7-alpha perf stack; reproduces on v0.6.0 — pre-existing, not caused by the 0.7 branches.
Symptom
Once offered load exceeds the drain ceiling long enough for the ready backlog to fragment across many sealed ring generations, drain rate collapses and NEVER recovers, even though workers are idle-capable and the backlog is large:
v0.6.0 @ 8,000/s offered, W=128 (4CPU-capped postgres): enqueue holds 7,998/s, backlog grows 77k→350k, late-window drain wedges at 3,358/s (healthy ceiling is >5k at this shape).
Mechanism (verified with a deterministic seeder + A/B on both versions)
claim_ready_runtime's row-claim CTE filters to exactly ONE (ready_slot, ready_generation) per call — a claim cannot span generation boundaries. v_claim_limit (up to 512) is computed from the lane's whole outstanding, but the generation filter truncates the actual yield to the current generation's slice of the lane. With a backlog interleaved across N generations, yield-per-call ≈ slice size (measured: identical ~10/call on v0.6.0 and the ledger branch with burst-10 seeding across 15 generations). Under load, rotation stalls once all slots hold undrained rows (busy-check), pinning the fragmentation in place — the system stays in the low-yield regime until the entire backlog drains at the wedged rate.
There may additionally be a zero-yield call-storm component (measured mean yield ~10/call implies many empty routing/probe calls when slice arithmetic predicts ~500-row slices) — needs distribution-level measurement, not just pgss means.
Why it matters
A transient overload burst (or a slow consumer deploy) that lets backlog fragment turns into a persistent ~60% throughput loss with no self-healing. Backpressure (#341) reduces entry probability but doesn't remove the cliff; adaptive claimers (#380) don't either (yield is per-call, not per-claimer).
Candidate fix
Let one claim call span multiple CONTIGUOUS generations of the same lane up to v_claim_limit — raises the wedge floor to the true batch cap. This is a claim-CTE redesign (routing + attempt/receipt batch writes are keyed by (slot, generation)) and needs TLA+ model updates (AwaSegmentedStorage claim actions) — not a contained patch.
Evidence: bench repo results/2026-07-11-awa-07-alpha-gate + wedge cells (wedge_old/wedge_new); reproduction seeder described in PR #415's investigation thread.
Measured magnitudes (orphan-swept cells, W=128, 4CPU-capped postgres): at 8,000/s offered (past the ~6.5k drain knee), v0.6.0 drains 7,199/s (backlog →231k, p99 ~18.6s) and the 0.7 ledger branch drains 7,641/s (backlog →115k, p99 ~7.9s). Drain stays near the healthy ceiling; the dominant overload symptom is unbounded backlog + multi-second latency (the #341 backpressure gap). The single-generation yield ceiling below is real (deterministic seeder, identical on both versions) but requires deeper fragmentation than these shapes produce — it is a latent ceiling that mainly slows recovery from extreme backlogs.
Found while gating the 0.7-alpha perf stack; reproduces on v0.6.0 — pre-existing, not caused by the 0.7 branches.
Symptom
Once offered load exceeds the drain ceiling long enough for the ready backlog to fragment across many sealed ring generations, drain rate collapses and NEVER recovers, even though workers are idle-capable and the backlog is large:
Mechanism (verified with a deterministic seeder + A/B on both versions)
claim_ready_runtime's row-claim CTE filters to exactly ONE (ready_slot, ready_generation) per call — a claim cannot span generation boundaries.v_claim_limit(up to 512) is computed from the lane's whole outstanding, but the generation filter truncates the actual yield to the current generation's slice of the lane. With a backlog interleaved across N generations, yield-per-call ≈ slice size (measured: identical ~10/call on v0.6.0 and the ledger branch with burst-10 seeding across 15 generations). Under load, rotation stalls once all slots hold undrained rows (busy-check), pinning the fragmentation in place — the system stays in the low-yield regime until the entire backlog drains at the wedged rate.There may additionally be a zero-yield call-storm component (measured mean yield ~10/call implies many empty routing/probe calls when slice arithmetic predicts ~500-row slices) — needs distribution-level measurement, not just pgss means.
Why it matters
A transient overload burst (or a slow consumer deploy) that lets backlog fragment turns into a persistent ~60% throughput loss with no self-healing. Backpressure (#341) reduces entry probability but doesn't remove the cliff; adaptive claimers (#380) don't either (yield is per-call, not per-claimer).
Candidate fix
Let one claim call span multiple CONTIGUOUS generations of the same lane up to
v_claim_limit— raises the wedge floor to the true batch cap. This is a claim-CTE redesign (routing + attempt/receipt batch writes are keyed by (slot, generation)) and needs TLA+ model updates (AwaSegmentedStorage claim actions) — not a contained patch.Evidence: bench repo
results/2026-07-11-awa-07-alpha-gate+ wedge cells (wedge_old/wedge_new); reproduction seeder described in PR #415's investigation thread.Milestone: 0.7 (correctness/operational).