broker,cli,chaos: lease-election races and live range failover (#223) - #236
Conversation
|
Bugbot is not enabled for your account, so this pull request was not reviewed. Enable Bugbot in the Cursor dashboard to get automatic reviews on future PRs. |
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 2ea8914cea
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| emit_leader_config leader > /dev/null | ||
| { | ||
| cat "$WORKDIR/data-leader-leader.yaml" |
There was a problem hiding this comment.
When this leased config is built by copying emit_leader_config, it also copies fencing_epoch: $FENCING_EPOCH (default 18) into a range whose metadata epoch starts at 0, so the first acquired leases are 1, 2, etc. The broker adoption path is monotonic (adopt_fencing_epoch/MetaFencingEpoch::set ignore lower epochs), so those metadata grants never update the broker and the scenario continues producing with the fixed epoch from emit_client_config; the live-chaos gate can therefore pass without actually exercising metadata-driven fencing/failover. Seed leased nodes from the metadata grant (or start the fixed epoch below it and generate client configs from the granted epoch).
Useful? React with 👍 / 👎.
There was a problem hiding this comment.
Fixed, and the finding was exactly right: with the floor at 18 and metadata minting from 1, the gate passed without metadata-driven fencing ever running. Three changes, because the pull was deeper than the config line: (1) leased leader configs (and the promoted-follower config) now rewrite fencing_epoch to 0, so the floor sits below every real grant; (2) clients are generated FROM the granted epoch — the producer uses emit_client_config_at_epoch $EPOCH_BEFORE, post-failover verify uses $EPOCH_AFTER; (3) followers — which validate replica appends against their static epoch and have no lease agent — are started at the epoch metadata is about to mint (asserted loudly) and the surviving follower is restarted at the new epoch after failover, with a comment saying plainly that the harness is standing in for the not-yet-built applied-state watcher. Together with the node-side change in #234 (lease-configured brokers start fenced), the scenario now cannot pass without the metadata plane actually driving leadership.
| # --- the dead leader must not be able to write again ------------------------ | ||
| # Restarting it against its old directory is the realistic operator mistake: | ||
| # the process comes back believing it still leads. | ||
| OLD=$(start_leader_with_lease "$LEADER_ID" restarted || true) |
There was a problem hiding this comment.
At this point NEW is still bound to native_addr/replica_addr 0, and start_leader_with_lease ... restarted emits the same listen addresses, so the restarted old leader fails to bind; the || true masks that failure, and the following assert_fenced_produce "$(native_addr)" talks to the new leader instead. This means the stale-old-leader assertion is not being exercised; stop/move the new leader or give the old process a distinct address and fail if it does not actually start.
Useful? React with 👍 / 👎.
There was a problem hiding this comment.
Fixed — see the cubic thread on the same line: the restarted old leader now binds distinct ports via start_fenced_old_leader, the || true is gone, readiness-never-arrives is asserted via await_not_ready on its own endpoint, and the fenced-produce assertion targets its own native address rather than the promoted follower's.
![cubic-dev-ai[bot]](https://avatars.githubusercontent.com/in/1082092?s=60&v=4){kind=small}
There was a problem hiding this comment.
All reported issues were addressed across 5 files
Reply with feedback, questions, or to request a fix.
Re-trigger cubic
0a927cd to
f044ca9
Compare
2ea8914 to
c1da48f
Compare
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: c1da48f1f5
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| # Promote follower 1 by starting it as a lease-driven leader against the same | ||
| # data directory it already replicated into. Its lease agent must win the range | ||
| # once the dead leader's lease lapses — which is the whole point of #223. | ||
| NEW=$(start_promoted_follower 1 "$LEADER_ID") |
There was a problem hiding this comment.
In this workflow, produce --durability quorum can acknowledge a batch after the leader plus either follower has fsynced it; it does not prove follower 1 has the $ACKED floor. If follower 2 was the fast quorum member for any final batches, promoting follower 1 here establishes the boundary from follower1+follower2 below $ACKED, and the subsequent verify --expect-at-least "$ACKED" fails even though the old quorum behaved correctly. Wait for follower 1 (or all replicas) to report $ACKED before killing/promoting, or promote a replica known to contain that floor.
Useful? React with 👍 / 👎.
There was a problem hiding this comment.
Fixed. The scenario no longer promotes a fixed follower: after the kill it reads both followers' durable offsets from their metrics endpoints and promotes the one holding the acknowledged floor — for any given final batch the other follower may legitimately have been the fast quorum member, exactly as you describe. The post-failover verify is also now await_verified_floor, retried until the new leader's replication stream catches the lagging follower up, since immediately after promotion the boundary a quorum can PROVE may still sit below the floor.
f044ca9 to
028ce54
Compare
c1da48f to
9954d9e
Compare
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 9954d9eccf
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| local id="$1" pid | ||
| local cfg="$WORKDIR/data-leader-restarted.yaml" | ||
| { | ||
| sed -e 's/^fencing_epoch: .*/fencing_epoch: 0/' \ |
There was a problem hiding this comment.
For the restarted-old-leader case, this rewrites the broker's held epoch to 0 even though the scenario later sends $EPOCH_BEFORE to prove the old grant is stale. Because lease-configured brokers start with an inactive metadata view, await_not_ready can return on that initial state and assert_fenced_produce can pass from an epoch mismatch with held epoch 0, without the lease agent ever observing the rival grant. Restart with the old granted epoch and wait for the metadata view to reflect the rival lease before asserting the stale produce is fenced.
Useful? React with 👍 / 👎.
There was a problem hiding this comment.
Fixed. start_fenced_old_leader now takes the old granted epoch and seeds it verbatim — exactly what a restarted operator process would carry — and the scenario waits (await_metric_at_least on vtop_broker_meta_fencing_epoch) until the restarted node's lease agent has actually OBSERVED the rival grant before probing. Only then does it assert /readyz stays closed and the $EPOCH_BEFORE produce is refused, so the refusal being proven is the one fencing provides — the broker holds the old epoch, its metadata view shows the rival's — rather than the trivial mismatch a zeroed epoch would have produced without the lease machinery ever running.
| # follower up: immediately after promotion the boundary a quorum can PROVE may | ||
| # still sit below the floor. | ||
| VERIFY_CFG="$(emit_client_config_at_epoch "$EPOCH_AFTER")" | ||
| await_verified_floor "$VERIFY_CFG" "$(native_addr)" "$ACKED" |
There was a problem hiding this comment.
When the final acknowledged quorum was the old leader plus the follower chosen for promotion, the remaining follower can still be below $ACKED. This revision's max-offset selection still starts the promoted leader with only that remaining follower, so verified promotion publishes the 2-of-2 floor below $ACKED; after this point the harness only retries verify and performs no produce/recovery action that would advance ClusterCommittedOffset, so the scenario can time out despite correct quorum behavior. Wait until both followers report the acknowledged floor, or explicitly recover and commit it, before asserting readability.
Useful? React with 👍 / 👎.
There was a problem hiding this comment.
Fixed, and your premise was right: nothing after promotion advanced ClusterCommittedOffset, so when the restarted follower was the lagging one the 2-of-2 floor sat below $ACKED and the retry loop had nothing to wait FOR. The scenario now resumes traffic on the new leader before verifying: one quorum-acked batch at the new epoch, with --first-sequence $RECORDS so producer idempotency cannot dedupe it into a no-op against records the interrupted producer already sent. Quorum-acking that batch forces the replication stream to push the backlog to the lagging follower, which is what moves the proven boundary past the pre-kill floor; the retried verify then has a guaranteed arrival rather than a hope. (Waiting for both followers to hold $ACKED before the kill was the alternative, but it would undo the mid-flight property this revision added.)
028ce54 to
dcbc9e2
Compare
9954d9e to
1b63b31
Compare
|
You have reached your Codex usage limits for code reviews. You can see your limits in the Codex usage dashboard. |
bab96ce to
8e2e4d2
Compare
Fifth slice of #223. The mechanism exists; this is the evidence it works. `lease_election_races.rs` decides the races deterministically against the real metadata state machine, with explicit `issued_at_ms` values — "two candidates at the same instant" means two commands carrying the same timestamp applied in log order, which is what Raft delivers to every replica. Nothing here depends on wall-clock timing or on tests racing each other. The five cases are the ones a timing-based design gets wrong. Two simultaneous candidates: exactly one wins, because the loser's CAS token is stale by the time its command applies. A clock-skewed candidate an hour fast: it takes the range early — that is the disruption the design admits to — and the epoch it mints still fences the previous holder, whose broker is then refused on the data path. A late renewal from a displaced holder, which is what a partitioned leader produces when its heartbeats land after the range has moved: refused. A holder that keeps renewing: never displaced, and its epoch never churns, because a new epoch per heartbeat would fence a leader against its own in-flight produce. And re-acquisition by the current holder mints exactly one epoch, so an agent that retried cannot ratchet itself out of its own range. Together they state the claim the design rests on: expiry is liveness, the fencing epoch is safety. `09-range-leader-failover.sh` runs the same thing on real processes. Every earlier data-plane scenario validated durability — kill the leader, and what was acknowledged survives — but none validated failover, because until now there was nothing to fail over to and the range simply stopped. This kills a leader under sustained quorum produce, restarts a follower as a lease-driven leader over the data it already replicated, and asserts the follower takes the range at a higher epoch, that every acknowledged record is still readable byte-exactly, that the restarted old leader is refused under its stale epoch, and that every surviving artifact verifies offline. Supporting: `vtopctl meta range-lease` exposes the linearizable read so a scenario (or an operator mid-incident) can see who holds a range and until when, and `vtopctl meta create-topic` fills the gap that made a range unleasable from the CLI at all.
1b63b31 to
42f00bd
Compare
|
You have reached your Codex usage limits for code reviews. You can see your limits in the Codex usage dashboard. |
…io 09) Scenario 09 — the range-leader-failover test, and the only live coverage of the subsystem #239 and #240 will modify — has been failing since it landed in #236. Nothing caught it because the live-chaos harness never runs in CI. Three defects, all the same root confusion: a producer's SEQUENCE and a range's OFFSET are different coordinates, and they coincide only when one producer wrote every record in the range contiguously from sequence 0. A producer resuming after a failover satisfies none of that. The scenario resumed the interrupted producer at `--first-sequence $RECORDS`, reasoning that starting past the interrupted range avoids idempotent dedupe. It does — and it also creates a sequence gap, which the broker refuses, as it must: gap-free sequencing is what makes an idempotent producer idempotent. The assertion could only pass when the "mid-flight" kill landed after the producer had already written all $RECORDS records, i.e. when the thing the scenario exists to test did not happen. The producer now bumps its PRODUCER EPOCH instead, which opens a fresh sequence space (state is keyed on `(producer_id, producer_epoch)`) and fences the pre-failover session. That is the mechanism a real client has: its producer id is pinned to the authenticated principal, so it cannot present a different identity, and promotion truncates to the verified quorum floor, so it cannot know where its old sequence space now ends. `produce` asserted `committed_next_offset == first_sequence + acked + count`, which is the same conflation on the client side — it failed a correct broker whenever the range already held records. It now anchors on the first observed offset and asserts each ack advances the range by exactly the records acked, which is the invariant actually being claimed, without assuming the range started empty or that this producer owns it. `verify` reconstructs expected record content from the offset and checked every record to the high watermark, so a range containing anything this producer did not write was unverifiable. It takes `--verify-content-through` (default unbounded, so every existing caller is unchanged); scenario 09 bounds it at the acknowledged floor, which is exactly the claim being made. Structure — offset contiguity and the high watermark — is still checked throughout. The post-failover produce is also retried to a deadline: the follower was restarted moments earlier and the leader's replication stream to it is established asynchronously, so a first attempt finding zero followers durable is the stream still connecting, not a durability failure. A single attempt made the scenario depend on winning that race. Verified: scenario 09 passes end to end. Full suite is 8 of 13 passing. Of the five failures, four are macOS-only limitations — 05, 05b and 06 require `unshare`, 07 requires a Linux clock shim — and all four are expected to run on CI's ubuntu runners. The fifth, scenario 10, fails identically on a clean tree ("joint membership with 0 configs cannot be stored in MetaMembership" at init) and is tracked separately; it blocks enabling this suite in CI.
The live-chaos harness drives real processes over real TLS — metadata Raft, replication, fencing, promotion, failover — and it has never run in CI. No workflow referenced it. It executed only when someone ran it by hand. That gap is not theoretical. Scenario 09 sat red on main from #236, scenario 10 from #237, a data node presented a metadata node's certificate to the admin endpoint across three issues, and scenario 01 failed on a scheduling race that appeared and vanished between runs. Every one of those was found by running the suite manually this week, not by CI, and each had been on main for weeks. The harness was not linted either: the `shell` filter listed `docker/*.sh` and `docker/tests/**`, so editing `lib.sh` — which all thirteen scenarios source — triggered nothing at all. Both are now covered: * `shell` gains `scripts/live-chaos/**`, and a second shellcheck step lints the harness with `-x -P scripts/live-chaos/scenarios` so the scenarios' dynamic `source` resolves. Without `-P`, every scenario reports SC1091 plus spurious "may not be assigned" warnings for variables lib.sh does define — noise that trains people to ignore the job. The tree is clean under it. * A new `chaos` filter and `live-chaos` job run the suite. Scoped, not blanket: only the harness itself and the four crates whose binaries it executes can change a scenario's outcome, so a docs or dashboard edit does not pay for a cluster. Debug binaries, because these scenarios assert correctness rather than throughput. Unprivileged user and mount namespaces are enabled and then PROBED. Ubuntu 24.04 restricts unprivileged userns by default, and scenarios 05, 05b, and 06 need it. The probe does not skip on failure: these scenarios fail with a remediation message rather than skipping, so a runner without namespaces turns the job red honestly instead of reporting success for three scenarios that never ran — which is exactly the silent gap this job exists to close. Per-scenario logs upload on failure. A failing scenario names a log file, and without the artifact that diagnosis dies with the runner. Enabled only now because the suite had to be green first: #251, #252, and #253 fixed the three real failures. Local state is 9 of 13 passing, with the four remaining failures all requiring Linux facilities absent on macOS — `unshare` for 05, 05b, 06 and a clock shim for 07 — which is what this job exists to exercise. Scenario 01 was run five times after the race fix: 5 of 5.
…io 09) (#251) Scenario 09 — the range-leader-failover test, and the only live coverage of the subsystem #239 and #240 will modify — has been failing since it landed in #236. Nothing caught it because the live-chaos harness never runs in CI. Three defects, all the same root confusion: a producer's SEQUENCE and a range's OFFSET are different coordinates, and they coincide only when one producer wrote every record in the range contiguously from sequence 0. A producer resuming after a failover satisfies none of that. The scenario resumed the interrupted producer at `--first-sequence $RECORDS`, reasoning that starting past the interrupted range avoids idempotent dedupe. It does — and it also creates a sequence gap, which the broker refuses, as it must: gap-free sequencing is what makes an idempotent producer idempotent. The assertion could only pass when the "mid-flight" kill landed after the producer had already written all $RECORDS records, i.e. when the thing the scenario exists to test did not happen. The producer now bumps its PRODUCER EPOCH instead, which opens a fresh sequence space (state is keyed on `(producer_id, producer_epoch)`) and fences the pre-failover session. That is the mechanism a real client has: its producer id is pinned to the authenticated principal, so it cannot present a different identity, and promotion truncates to the verified quorum floor, so it cannot know where its old sequence space now ends. `produce` asserted `committed_next_offset == first_sequence + acked + count`, which is the same conflation on the client side — it failed a correct broker whenever the range already held records. It now anchors on the first observed offset and asserts each ack advances the range by exactly the records acked, which is the invariant actually being claimed, without assuming the range started empty or that this producer owns it. `verify` reconstructs expected record content from the offset and checked every record to the high watermark, so a range containing anything this producer did not write was unverifiable. It takes `--verify-content-through` (default unbounded, so every existing caller is unchanged); scenario 09 bounds it at the acknowledged floor, which is exactly the claim being made. Structure — offset contiguity and the high watermark — is still checked throughout. The post-failover produce is also retried to a deadline: the follower was restarted moments earlier and the leader's replication stream to it is established asynchronously, so a first attempt finding zero followers durable is the stream still connecting, not a durability failure. A single attempt made the scenario depend on winning that race. Verified: scenario 09 passes end to end. Full suite is 8 of 13 passing. Of the five failures, four are macOS-only limitations — 05, 05b and 06 require `unshare`, 07 requires a Linux clock shim — and all four are expected to run on CI's ubuntu runners. The fifth, scenario 10, fails identically on a clean tree ("joint membership with 0 configs cannot be stored in MetaMembership" at init) and is tracked separately; it blocks enabling this suite in CI.
The live-chaos harness drives real processes over real TLS — metadata Raft, replication, fencing, promotion, failover — and it has never run in CI. No workflow referenced it. It executed only when someone ran it by hand. That gap is not theoretical. Scenario 09 sat red on main from #236, scenario 10 from #237, a data node presented a metadata node's certificate to the admin endpoint across three issues, and scenario 01 failed on a scheduling race that appeared and vanished between runs. Every one of those was found by running the suite manually this week, not by CI, and each had been on main for weeks. The harness was not linted either: the `shell` filter listed `docker/*.sh` and `docker/tests/**`, so editing `lib.sh` — which all thirteen scenarios source — triggered nothing at all. Both are now covered: * `shell` gains `scripts/live-chaos/**`, and a second shellcheck step lints the harness with `-x -P scripts/live-chaos/scenarios` so the scenarios' dynamic `source` resolves. Without `-P`, every scenario reports SC1091 plus spurious "may not be assigned" warnings for variables lib.sh does define — noise that trains people to ignore the job. The tree is clean under it. * A new `chaos` filter and `live-chaos` job run the suite. Scoped, not blanket: only the harness itself and the four crates whose binaries it executes can change a scenario's outcome, so a docs or dashboard edit does not pay for a cluster. Debug binaries, because these scenarios assert correctness rather than throughput. Unprivileged user and mount namespaces are enabled and then PROBED. Ubuntu 24.04 restricts unprivileged userns by default, and scenarios 05, 05b, and 06 need it. The probe does not skip on failure: these scenarios fail with a remediation message rather than skipping, so a runner without namespaces turns the job red honestly instead of reporting success for three scenarios that never ran — which is exactly the silent gap this job exists to close. Per-scenario logs upload on failure. A failing scenario names a log file, and without the artifact that diagnosis dies with the runner. Enabled only now because the suite had to be green first: #251, #252, and #253 fixed the three real failures. Local state is 9 of 13 passing, with the four remaining failures all requiring Linux facilities absent on macOS — `unshare` for 05, 05b, 06 and a clock shim for 07 — which is what this job exists to exercise. Scenario 01 was run five times after the race fix: 5 of 5.
* ci: run and lint the live-chaos harness, path-gated The live-chaos harness drives real processes over real TLS — metadata Raft, replication, fencing, promotion, failover — and it has never run in CI. No workflow referenced it. It executed only when someone ran it by hand. That gap is not theoretical. Scenario 09 sat red on main from #236, scenario 10 from #237, a data node presented a metadata node's certificate to the admin endpoint across three issues, and scenario 01 failed on a scheduling race that appeared and vanished between runs. Every one of those was found by running the suite manually this week, not by CI, and each had been on main for weeks. The harness was not linted either: the `shell` filter listed `docker/*.sh` and `docker/tests/**`, so editing `lib.sh` — which all thirteen scenarios source — triggered nothing at all. Both are now covered: * `shell` gains `scripts/live-chaos/**`, and a second shellcheck step lints the harness with `-x -P scripts/live-chaos/scenarios` so the scenarios' dynamic `source` resolves. Without `-P`, every scenario reports SC1091 plus spurious "may not be assigned" warnings for variables lib.sh does define — noise that trains people to ignore the job. The tree is clean under it. * A new `chaos` filter and `live-chaos` job run the suite. Scoped, not blanket: only the harness itself and the four crates whose binaries it executes can change a scenario's outcome, so a docs or dashboard edit does not pay for a cluster. Debug binaries, because these scenarios assert correctness rather than throughput. Unprivileged user and mount namespaces are enabled and then PROBED. Ubuntu 24.04 restricts unprivileged userns by default, and scenarios 05, 05b, and 06 need it. The probe does not skip on failure: these scenarios fail with a remediation message rather than skipping, so a runner without namespaces turns the job red honestly instead of reporting success for three scenarios that never ran — which is exactly the silent gap this job exists to close. Per-scenario logs upload on failure. A failing scenario names a log file, and without the artifact that diagnosis dies with the runner. Enabled only now because the suite had to be green first: #251, #252, and #253 fixed the three real failures. Local state is 9 of 13 passing, with the four remaining failures all requiring Linux facilities absent on macOS — `unshare` for 05, 05b, 06 and a clock shim for 07 — which is what this job exists to exercise. Scenario 01 was run five times after the race fix: 5 of 5. * ci: run CI on every pull request, not only those targeting main `pull_request: branches: [main]` meant a stacked PR ran no CI whatsoever. This project stacks slices — each bases on the one below — so those PRs carried a single check, an AI reviewer's comment, and nothing else. They could be reviewed, approved, and merged with no build, no tests, and no lint. Nor did merging the parent repair it. Retargeting a PR's base fires the `edited` activity type, which is not in the default `pull_request` event set, so the retargeted PR did not reliably re-trigger CI either. The first real signal arrived on push to main — after the merge, on the branch that is supposed to stay green. The path filters already scope each run, so the added cost is only the jobs a stacked slice actually needs. This is the same shape as the gap the previous commit closes: a check that exists and is trusted, but never runs on the change it is meant to guard. * ci: make the live-chaos log artifact actually contain logs The upload step collected nothing. Scenarios delete workdirs they generated themselves, so by the time the step ran the `/tmp/vtop-chaos.*` tree was gone — and `if-no-files-found: ignore` turned that into a silent success. The first run confirmed it: the job passed and produced no artifact at all. A diagnostic that appears to exist and is absent exactly when it is needed is worse than none, because it stops anyone from looking for a real one. Supplying CHAOS_WORKDIR fixes both halves: the logs land in a known path, and because the harness only cleans up directories it generated itself, providing one suppresses the cleanup. `if-no-files-found` becomes `error`, so a future break in this wiring fails the job instead of quietly returning to collecting nothing. Upload is now `if: failure()` — the logs are for diagnosing a failed run, and uploading them on every green run is storage for nobody. Verified by running a scenario with CHAOS_WORKDIR set and confirming the log tree survives. * ci: widen the chaos filter to the real dependency closure, and gate the upload correctly Two review findings, both correct. The chaos filter named four crates and asserted that nothing else could change a scenario's outcome. That assertion was false. The two binaries this job runs pull in every workspace crate: cargo tree -p vtop-node --edges normal → broker, log, meta, observe, protocol cargo tree -p vtop-cli --no-default-features --edges normal → adapters, broker, core, log, meta, observe, protocol, state, upload The omissions were not marginal. vtop-log owns the segment format the durability and recovery scenarios assert on, and vtop-protocol owns the produce/fetch wire every scenario speaks — either could have broken the suite with this job never running, which is precisely the hole this job exists to close. The filter is now `crates/**`, derived from the closure rather than from a guess about it. That means the suite runs on most Rust changes. That is the honest cost of it being the only thing validating replication and fencing against real processes; a cheaper gate with a hole in it is what was just removed. The log upload was gated on `failure()`, which is also true when the namespace probe or the build failed. In those cases the suite never reached a scenario, so no logs exist, and `if-no-files-found: error` would stack a spurious "no files found" on top of the actual cause. It now keys on the run step's own outcome, so the empty-artifact signal only fires once a run was actually attempted — which is the only situation in which it means anything.
1 participant
Fifth and final slice of #223, stacked on #235. The mechanism exists; this is the evidence it works.
Deterministic races
lease_election_races.rsdecides the races against the real metadata state machine with explicitissued_at_msvalues. "Two candidates at the same instant" means two commands carrying the same timestamp applied in log order — which is what Raft delivers to every replica. Nothing depends on wall-clock timing or on tests racing each other.The five cases are the ones a timing-based design gets wrong:
Together they state the claim the design rests on: expiry is liveness, the fencing epoch is safety.
Live failover
09-range-leader-failover.shruns the same thing on real processes. Every earlier data-plane scenario validated durability — kill the leader, and what was acknowledged survives — but none validated failover, because until now there was nothing to fail over to and the range simply stopped.It kills a leader under sustained quorum produce, restarts a follower as a lease-driven leader over the data it already replicated, and asserts:
Supporting CLI
vtopctl meta range-leaseexposes the linearizable read, so a scenario — or an operator mid-incident — can see who holds a range and until when.vtopctl meta create-topicfills the gap that made a range unleasable from the CLI at all.Refs #223. With this merged, #223's acceptance is met.
Summary by cubic
Deterministic lease-election races and a live failover scenario prove lease-driven promotion and fencing work end to end. Adds
vtopctlcommands to create a topic and read a range lease, fulfilling #223.vtop-brokeraddslease_election_races.rswith five deterministic cases using explicitissued_at_ms, including data-path fencing of stale epochs.09-range-leader-failover.shkills a leader under sustained quorum produce, promotes the follower with the highest committed offset via the lease, asserts a higher epoch, intact acknowledged data, a restarted old leader that stays unready and is fenced, and verifies artifacts; lease helpers inscripts/live-chaos/lib.sh; lease env vars inLIVE_CHAOS_VALIDATION.md.vtopctl meta range-leasereturns a linearizable view (found, holder, fencing epoch, deadline, range generation, applied index; with JSON output);vtopctl meta create-topiccreates a topic and root range for leasing.Written for commit 42f00bd. Summary will update on new commits.