test(ci): give nightly flake gates contention margin; add pg18 + py3.13/3.14

.github/workflows/ci.yml

@@ -109,7 +109,7 @@ jobs:
         run: cargo build --workspace
 
   rust-test:
-    name: Rust tests (${{ matrix.shard }})
+    name: Rust tests (pg${{ matrix.postgres }}, ${{ matrix.shard }})
     needs: [rust-build]
     if: github.event_name != 'pull_request' || contains(github.event.pull_request.labels.*.name, 'full-ci')
     runs-on: ubuntu-latest
@@ -123,6 +123,11 @@ jobs:
     strategy:
       fail-fast: false
       matrix:
+        # Postgres majors we support. The schema, its partitioning, and the
+        # planner behaviour the claim path depends on are all version-
+        # sensitive, so the whole sharded suite runs on each major rather
+        # than a smoke subset.
+        postgres: ["17", "18"]
         shard:
           - migrations-1
           - migrations-2
@@ -132,7 +137,7 @@ jobs:
           - rest
     services:
       postgres:
-        image: postgres:17-alpine
+        image: postgres:${{ matrix.postgres }}-alpine
         env:
           POSTGRES_PASSWORD: postgres
           POSTGRES_DB: awa_test
@@ -164,14 +169,18 @@ jobs:
   # canonical, so this leg re-runs it with the queue-storage engine active on a
   # fresh database (a separate DB avoids cross-engine job_unique_claims carryover).
   rust-test-queue-storage:
-    name: Rust tests (queue_storage)
+    name: Rust tests (queue_storage, pg${{ matrix.postgres }})
     needs: [rust-build]
     if: github.event_name != 'pull_request' || contains(github.event.pull_request.labels.*.name, 'full-ci')
     runs-on: ubuntu-latest
     timeout-minutes: 30
+    strategy:
+      fail-fast: false
+      matrix:
+        postgres: ["17", "18"]
     services:
       postgres:
-        image: postgres:17-alpine
+        image: postgres:${{ matrix.postgres }}-alpine
         env:
           POSTGRES_PASSWORD: postgres
           POSTGRES_DB: awa_test
@@ -274,10 +283,20 @@ jobs:
 
   # ─── Python (awa-python) ────────────────────────────────
   python-build-test:
-    name: Python build + test
+    name: Python build + test (py${{ matrix.python }})
     if: github.event_name != 'pull_request' || contains(github.event.pull_request.labels.*.name, 'full-ci')
     runs-on: ubuntu-latest
     timeout-minutes: 60
+    strategy:
+      fail-fast: false
+      matrix:
+        # The newest two interpreters only. The wheel is abi3-py310 so one
+        # artifact covers the whole 3.10+ support window, and abi3 keeps the
+        # C API stable across it; these legs exercise the SDK's asyncio
+        # integration (pyo3-async-runtimes), which is where version skew
+        # actually lands — and it lands on the newest releases. When a new
+        # CPython ships, add it here and drop the oldest.
+        python: ["3.13", "3.14"]
     services:
       postgres:
         image: postgres:17-alpine
@@ -300,14 +319,15 @@ jobs:
       - uses: Swatinem/rust-cache@v2
         with:
           workspaces: "awa-python -> awa-python/target"
+          key: "py${{ matrix.python }}"
       - uses: actions/setup-python@v6
         with:
-          python-version: "3.12"
+          python-version: "${{ matrix.python }}"
       - name: Install uv
         uses: astral-sh/setup-uv@v8.2.0
       - name: Create venv and install deps
         run: |
-          uv venv .venv
+          uv venv .venv --python "${{ matrix.python }}"
           uv pip install maturin pytest pytest-asyncio asyncpg django greenlet psycopg psycopg-binary sqlalchemy
       - name: Build extension (maturin develop)
         run: .venv/bin/maturin develop

AGENTS.md

@@ -29,6 +29,59 @@ DATABASE_URL=postgres://postgres:test@localhost:15432/awa_test uv run pytest tes
 Read the redirected log and check the real exit status; do not trust a green
 exit code from a piped or backgrounded command.
 
+### Version matrix
+
+CI runs the sharded Rust suite and the queue-storage leg against **Postgres 17
+and 18** — both majors run the whole suite, because the schema, its
+partitioning, and the planner behaviour the claim path depends on are all
+version-sensitive.
+
+The Python suite runs on the **newest two CPython releases**. The support
+window is whatever `requires-python` says (3.10+), and the `abi3-py310` wheel
+covers all of it from one artifact, so the legs are not there to prove the
+ABI — they exercise the asyncio integration (`pyo3-async-runtimes`), which is
+where version skew actually lands. The `pyproject.toml` classifiers therefore
+track the support window, not the test matrix. When a new CPython ships, add it
+to the matrix and drop the oldest.
+
+### Contention-sensitive assertions in the nightly suites
+
+The nightly chaos and benchmark suites run on shared runners whose CPU
+allocation varies run to run. Three assertion shapes are sensitive to that and
+must go through `awa/tests/ci_timing.rs` rather than hard-coding a bound:
+
+| Shape | Helper | Failure if too tight |
+| --- | --- | --- |
+| Wall-clock wait for a state | `scaled_timeout` | Spurious timeout |
+| `heartbeat_staleness` on a chaos client | `scaled_staleness` | The runtime rescues a *live* attempt and the test sees a genuine duplicate completion — a margin bug that reads as a correctness bug |
+| "at least N of these happened" floor | `contention_floor` | Gate fires while every invariant it exists for is intact |
+
+All three only ever loosen a bound, and only when `CI` is set, so a local run
+keeps the strict values and a real regression still fails fast on a developer
+machine. `AWA_CHAOS_TIMEOUT_MULTIPLIER` overrides the factor (clamped to
+`>= 1.0`).
+
+A minimum-progress floor must sit below the *observed* operating point, not the
+nominal one. The receipt-plane rotation gate is the cautionary example (#399):
+its floor came from the 1s rotate interval's tick rate, but rotation is driven
+by the maintenance loop reaching a rotate decision, so the healthy steady state
+sat flush against the floor and the gate fired at 41-vs-45 with every
+architectural bound perfect.
+
+Measurement windows are *not* timeouts — `recv_until`'s duration defines what a
+benchmark samples, so it stays unscaled.
+
+### Connection budget in the Python suite
+
+`awa-python/tests/conftest.py` defaults test clients to a small pool and fails
+the session if backend count grows across it. Two conventions follow (#420):
+
+- A fixture that builds a client must `yield` it and close it in a `finally`,
+  never `return` it. Returning leaves pool teardown to GC timing, which parks
+  server connections for up to sqlx's 10-minute idle timeout.
+- Pass `max_connections` explicitly only when the test is *about* pool sizing.
+  Otherwise take the conftest default, so one test cannot starve the next.
+
 ## Schema Migrations
 
 Migrations are forward-only and must stay rolling-upgrade compatible. Version

CHANGELOG.md

@@ -6,6 +6,14 @@ Notable changes between releases. Detailed migration notes for storage transitio
 
 ### Fixed
 
+- **Nightly flake gates now carry runner-contention margin ([#399](https://github.com/hardbyte/awa/issues/399), [#434](https://github.com/hardbyte/awa/issues/434)).** Four assertion shapes in the chaos and benchmark suites were tight enough that shared-runner CPU contention failed them while every invariant they exist for was intact, eroding the 14-consecutive-green-nightlies release gate. `awa/tests/ci_timing.rs` now holds the scaling for all of them, and it only ever loosens a bound, and only when `CI` is set:
+  - The mixed-fleet chaos test set `heartbeat_staleness` to 250ms against a 50ms heartbeat interval. Under contention a *live* worker's heartbeat missed that window, so the runtime correctly rescued a healthy attempt and the test saw a genuine duplicate completion — a margin problem that read as a correctness bug. Chaos clients scale the staleness window via `scaled_staleness` while leaving the heartbeat and rescue *intervals* at chaos cadence, so the rescue path is still exercised.
+  - That test's assert-after-drain step watched both completion streams for a fixed 250ms quiet window, the same race [#335](https://github.com/hardbyte/awa/issues/335) fixed in `test_weight_proportionality`: a duplicate arriving at 251ms was missed, and on a slow runner the window expired while work was still moving. It now waits for the queue itself to reach a terminal state — the authoritative drain signal — then drains both streams with `try_recv`, so duplicate detection no longer depends on wall-clock timing.
+  - The receipt-plane rotation floor (`>= 45` rotations in 180s) was derived from the 1s rotate interval's nominal tick rate, but rotation is driven by the maintenance loop reaching a rotate decision, so the healthy steady state is ~40-45 and the floor had no margin. It fired at 41 with zero dead rows in every claim and closure partition. The floor is now divided by the contention multiplier, keeping a genuine pinned-ring floor (a pinned ring shows ~0) with ~3x margin.
+  - `scheduling_benchmark_test.rs` had no contention scaling at all, so its `wait_for_leader`/`wait_for_dispatch` readiness gates and post-handler completion waits were hard 5s/30s deadlines. They now scale. Measurement windows (`recv_until`) deliberately do not — their duration defines what the benchmark samples.
+
+- **Python suite no longer exhausts the server's connection slots ([#420](https://github.com/hardbyte/awa/issues/420)).** Late in a full run, one or two tests ERRORed at *fixture setup* with `pool timed out while waiting for an open connection` — always ~15 minutes in, always rerun-clearable. `AsyncClient`/`Client` default to a 10-connection pool, and sqlx holds an idle connection for its 10-minute idle timeout, so pools from tests that finished minutes earlier kept slots parked while each new fixture asked for up to 10 more; when the server had none left, sqlx could not grow the pool and its 30s acquire timeout expired. The victim was therefore whichever test came next, not the test at fault. A new `awa-python/tests/conftest.py` caps what tests ask for, and two fixtures that built a client and `return`ed it — leaving pool teardown to GC timing rather than closing it — now yield and close deterministically. A session-scoped guard fails the run with a named backend count if a `return`-style fixture reappears. Verified by re-running the full suite against a deliberately constrained 30-connection server: the pre-fix run parked 11-19 of 30 backends on pools belonging to tests that had finished minutes earlier, and the post-fix run holds 3 and passes all 312.
+
 - **`awa.jobs` is receipt-aware and no longer pathologically slow under backlog ([#422](https://github.com/hardbyte/awa/issues/422), migration v044).** The SQL-compat view had two defects found during the 0.6.1→0.7 upgrade rehearsal. *Performance:* the available-branch filter spelled `lane_seq >= sequence_next_value(claims.seq_name)` inline against the partitioned `ready_entries` scan; `sequence_next_value` is a VOLATILE PL/pgSQL function, so the planner could neither use the predicate as an index bound nor cache it across rows — every ready child was scanned in full and each surviving row paid a dynamic-SQL catalog round-trip (measured ~90ms → ~6ms for the branch on a 20k-row backlog; the gap grows with un-pruned sealed generations). v044 materializes the per-lane cursors once per statement — the same shape the claim path has used since v027/v039 — and reading each cursor once is also a more consistent snapshot than per-row evaluation. *Correctness:* receipt claims that never materialized into `leases` (row-local `lease_claims`, and by default compact `lease_claim_batches`) were invisible, so `SELECT state, count(*) FROM awa.jobs` reported `running=0` while workers held live work. The view now carries the open-receipt legs the admin surface shipped in #410 (`admin::state_counts` / `queue_storage::open_receipt_running_claims_sql`), projecting the claim ledger's own attempt / claim-time / deadline values and anti-joined against closures, closure batches, materialised leases, and terminal/deferred/DLQ supersession — a job reports at most one state outside the brief post-commit cursor-lag window the claim protocol itself has (identical to the admin surfaces; it self-heals on the lane's next claim). Rows previously returned are unchanged; the running rows are additive. **No 0.6.x backport is required:** released 0.6.x workers have no schema-version gate at startup, so both upgrade orderings keep working — only a ≤0.6.x *migrator* refuses via the #392 fail-safe, which is correct (past-v043 migrations are applied by the 0.7 binary). Verified by the compat matrix against pinned release artifacts (`scripts/compat-matrix.sh` now runs a `forward-0.6.6` leg alongside 0.6.2/0.6.0/0.5.7) plus a released-binary enqueue→claim→complete rehearsal on a v044 database.
 
 - **`awa migrate --extract-to` wrote only the first 15 migrations.** The target path was built by substituting the migration description into the filename, so v017 — whose description contains `/` — resolved to a nested directory that does not exist. The command aborted there, leaving a partial extraction that looked plausible but silently omitted two thirds of the schema, breaking the documented external-runner workflow for every range reaching v017. Only path separators are now substituted, so every filename this tool has already published stays byte-identical — re-extracting into an existing directory cannot leave two files for the same version. `V17` and `V21` are the only names that change, and neither could previously be written at all. All paths are computed and checked for collisions before anything is written, and a write failure names the file. A test asserts every migration is extracted with byte-identical SQL.
@@ -25,6 +33,8 @@ Notable changes between releases. Detailed migration notes for storage transitio
 
 ### Added
 
+- **CI covers Postgres 18; the Python matrix moves to the newest two interpreters.** The sharded Rust suite and the queue-storage integration leg now run against Postgres 17 *and* 18 — the schema, its partitioning, and the planner behaviour the claim path depends on are all version-sensitive, so both majors run the whole suite rather than a smoke subset. (Postgres 18 was already in use by the telemetry-validation job and the Python nightly, unsystematically.) The Python suite moves from CPython 3.12 to 3.13 and 3.14: the `abi3-py310` wheel covers the whole `requires-python` window from one artifact, so the legs exist to exercise the asyncio integration where version skew actually lands rather than to re-prove the ABI on every supported interpreter. `pyproject.toml` now carries per-version classifiers for the support window it has always claimed.
+
 - **Migration atomicity and idempotency are now enforced by tests, not convention.** Four no-database unit tests assert the properties the single-transaction runner depends on: every migration step is transaction-safe (no `CONCURRENTLY`, `VACUUM`, or statement-level `BEGIN`/`COMMIT`/`ROLLBACK`/`SAVEPOINT`), every top-level `CREATE` is guarded so a replay is a no-op, every migration records its own version exactly once with `ON CONFLICT DO NOTHING`, and versions stay unique and ordered up to `CURRENT_VERSION`. Integration tests prove the runtime behaviour end-to-end: an injected mid-run DDL failure rolls back every earlier step (fresh install and upgrade alike) and is never visible to another session, replaying the full ordered set over a migrated database leaves a byte-identical catalog, and applying every migration twice is indistinguishable from a clean install. `awa migrate` itself is covered by a new CLI suite spanning the applied run, the rendered SQL, and the extracted files. `migrations::MIGRATION_LOCK_KEY` (and its Python counterpart `awa.migration_lock_key()`) is public so emitted SQL takes the same lock as the runner instead of a copy that could drift.
 
 ### Changed

awa-python/pyproject.toml

@@ -17,6 +17,11 @@ classifiers = [
     "Programming Language :: Rust",
     "Programming Language :: Python :: Implementation :: CPython",
     "Programming Language :: Python :: 3",
+    "Programming Language :: Python :: 3.10",
+    "Programming Language :: Python :: 3.11",
+    "Programming Language :: Python :: 3.12",
+    "Programming Language :: Python :: 3.13",
+    "Programming Language :: Python :: 3.14",
     "License :: OSI Approved :: MIT License",
     "License :: OSI Approved :: Apache Software License",
     "Topic :: Database",