Optimize vacuum/analyze

Cargo.toml

@@ -75,4 +75,6 @@ mimalloc = "0.1.52"
 
 [dev-dependencies]
 tempfile = "3"
+# Enable paused time for batch deadline tests
+tokio = { version = "1", features = ["test-util"] }
 toml = { version = "1", default-features = false, features = ["parse", "display", "serde"] }

bench/ec2/.gitignore

@@ -2,6 +2,9 @@
 *.pem
 state.env
 
+# Local AWS profile settings
+aws.local.env
+
 # copied-back perf/eBPF profiles (large)
 profiles/
 

bench/ec2/README.md

@@ -5,7 +5,9 @@ EC2 harness for comparing PG→destination replication engines, all `c8i.2xlarge
 cross-AZ latency. A **setup** =
 a base (always the source Postgres primary; plus ClickHouse for the CDC
 engines) + one **streamer** node that does the replication. The base is shared,
-so you swap the streamer while keeping the same source data.
+so you swap the streamer while keeping the same source data. The `ec2-bench`
+node that drives the load is part of every setup — see
+[Benchmark a setup](#benchmark-a-setup).
 
 | setup | streamer node | base | destination | notes |
 |-------|---------------|------|-------------|-------|
@@ -20,25 +22,45 @@ into each node folder for `deploy.sh` / `profile.sh` / the bench to read. The
 desired node set (streamer, clickhouse, bench runner) persists in
 `terraform/setup.auto.tfvars`, written by `stack.sh`. Post-boot setup stays
 shell: node folders keep `cloud-init.yaml` and, where needed, `deploy.sh` /
-`profile.sh` / `pre_down.sh`. Shared script helpers live in `aws-env.sh`
-(creds) and `lib.sh` (ssh/state.env). `*.pem`, `state.env` and terraform state
-are gitignored.
+`pre_down.sh`. Shared scripts sit alongside `stack.sh`: `aws-env.sh` (creds),
+`lib.sh` (ssh, state.env, readiness waits) and `profile.sh` (on-CPU capture for
+any streamer). `*.pem`, `state.env` and terraform state are gitignored.
+
+Record account and profile in `aws.local.env`, which stays gitignored and is
+required by `aws-env.sh`. Account check aborts when credentials resolve to a
+different account, then Terraform checks same account through
+`allowed_account_ids`:
+
+```bash
+echo 'BENCH_AWS_ACCOUNT=<account-id>' > aws.local.env
+echo 'BENCH_AWS_PROFILE=<p>' >> aws.local.env
+aws sso login --profile=<p>     # renew expired session
+```
+
+After the account check, `aws-env.sh` resolves that profile to session keys
+(`aws configure export-credentials`) and exports them, so terraform runs on
+exactly the creds the check validated — its Go SDK rejects a stale SSO token
+that the CLI would still serve from cache.
 
 ## stack.sh — the main interface
 
 ```bash
 cd bench/ec2
-./stack.sh up <setup>        # terraform apply (base + streamer), then deploy the streamer
-./stack.sh down              # tear down current streamer (base kept)
+./stack.sh up <setup>        # terraform apply (base + streamer + bench runner), then deploy
+./stack.sh down              # tear down current streamer (base + runner kept)
 ./stack.sh down --all        # terraform destroy (everything)
-./stack.sh bench up|down     # optional in-VPC bench-runner box (+ deploy)
+./stack.sh bench run <name>   # run the suite on the in-VPC runner → bench/results/<name>
+./stack.sh bench fetch <name> # re-copy a run's results off the runner
+./stack.sh bench up|down      # add/remove the runner box on its own
 ./stack.sh status            # list running project instances
 ```
 
 `terraform apply` is interactive — review the plan before confirming,
 especially on setup swaps (e.g. walshadow→pg destroys the ClickHouse node).
 Before a streamer node is destroyed or swapped, `stack.sh` copies any on-CPU
-profiles off it and runs its `pre_down.sh` hook. Terraform can also be driven
+profiles off it and runs its `pre_down.sh` hook. Start a capture with
+`./profile.sh <walshadow|peerdb|pg-standby> [secs]` just before a benchmark; it
+returns immediately and teardown copies the result into the node folder. Terraform can also be driven
 directly: `source aws-env.sh && terraform -chdir=terraform plan`. Knobs like
 `instance_type` / `az` / `my_ip` are variables (see `terraform/variables.tf`);
 `instance_type` is global — the AZ is picked from its offerings.
@@ -50,12 +72,18 @@ the source table at startup — so a second setup's run would disturb the first.
 ## Per-setup
 
 ### walshadow
-Build the image first (the deploy ships a locally-built image rather than building on the box):
+Build the image first (the deploy ships a locally-built image rather than building on the box).
+`PG_MAJOR` must match `ec2-source-pg` (`postgres:17`) — the shadow data dir comes
+from a BASE_BACKUP of the source, so PG 18 binaries cannot open it; `deploy.sh`
+compares the two and refuses to start on a mismatch:
 ```bash
-docker build -f docker/Dockerfile -t walshadow:local .   # from repo root
+docker build -f docker/Dockerfile --build-arg PG_MAJOR=17 -t walshadow:local .   # run from repository root
 ./stack.sh up walshadow                                   # source-pg + clickhouse + daemon
 ./stack.sh down                                           # daemon only (base kept)
 ```
+Building with podman works: it tags locally-built images `localhost/walshadow:local`,
+and `docker load` on the node keeps that prefix, so the deploys resolve whichever
+tag actually landed (`remote_image_tag` in `lib.sh`) instead of assuming `$IMAGE`.
 `deploy.sh` ships `walshadow:local` (`docker save | ssh | docker load`), writes
 `ch-config.toml` (ClickHouse private IP, `flush_timeout_ms`), and runs the daemon.
 
@@ -79,18 +107,43 @@ a read-only hot standby that streams WAL. Re-running takes a fresh base backup.
 
 ## Benchmark a setup
 
-`walshadow-ec2-bench` reads endpoints from the relevant `state.env`. Use
-`run_bench_suite.sh <name>` (at the bench crate root) to run all four benches into
-`bench/results/<name>/` (a gitignored dir, created on demand):
+The bench driver runs on `ec2-bench`, an in-VPC node that comes up with every
+setup. The driver's round trip lands in every commit→visible sample and caps
+the insert loop, so it has to sit next to the stack.
+
+`bench run` does the whole pass on the runner and copies the results into
+`bench/results/<name>/` (gitignored, created on demand; an existing name is
+refused). It brings the runner up and redeploys it first, so it works straight
+after `up`:
+```bash
+cd bench/ec2
+./stack.sh bench run walshadow-run                  # --dest defaults to clickhouse
+./stack.sh bench run pg-run --dest postgres         # physical standby (ec2-pg-standby)
+# sustained + interleaved for five minutes; interleaved-long is always 10 × 30s
+./stack.sh bench run walshadow-5min --run-secs 300
+./stack.sh bench fetch walshadow-5min               # re-copy after a dropped session
+```
+Extra flags pass through to `walshadow-ec2-bench`. Each shape runs as a child
+process, so one failure does not end the pass: its output is teed to
+`<shape>.txt` on the box with a `# FAILED` footer, and the suite exits non-zero
+listing what failed. `bench run` still fetches what landed. Every fetched run
+carries a `provenance.txt` (setup, instance type, AZ, repo commit).
+
+`--network` defaults to `private` and the on-box wrapper supplies `--state-dir`
+and `--results-dir`, so a single shape by hand is just:
 ```bash
-# CDC engines (walshadow / peerdb) → ClickHouse:
-../run_bench_suite.sh walshadow-run            # DEST defaults to clickhouse
-# physical standby:
-DEST=postgres ../run_bench_suite.sh pg-run     # reads ec2-pg-standby
+ssh -i terraform/walshadow-bench.pem ubuntu@<ec2-bench public ip>
+walshadow-ec2-bench --bench interleaved --xact-secs 150
 ```
+Running the binary from a workstation (`cargo run --release --bin
+walshadow-ec2-bench -- --network public …`) reaches the instances over the
+internet. Keep those runs for smoke tests — the numbers are not comparable with
+in-VPC ones.
 
 ## Notes
 - `c8i.2xlarge`s bill while running (~8× a t2.small) — `down` (or `down --all`) when idle.
+  A CDC setup is four of them: source-pg, clickhouse, streamer, bench runner.
+  `bench down` drops the runner alone, but any `up` brings it back.
 - SSH/Postgres/ClickHouse are open to the operator IP + VPC CIDR only; Postgres
   uses `trust` auth, so keep 5432 off `0.0.0.0/0`. The operator IP is captured
   at apply time — if yours changes, re-run `terraform apply` (or any `stack.sh`

bench/ec2/aws-env.sh

@@ -1,22 +1,81 @@
 #!/usr/bin/env bash
-# Source this before running aws commands:  source ../aws-env.sh
+# Load before running AWS commands: source ../aws-env.sh
+#
+# Refuse to run against unexpected account
+# Set account and profile in ignored aws.local.env
+#
+# Example:
+#   BENCH_AWS_ACCOUNT=<account-id>
+#   BENCH_AWS_PROFILE=<p>
+# Run aws sso login --profile=<p> when session expires
 #
 # Credential resolution, in order:
+#  - AWS_PROFILE, then BENCH_AWS_PROFILE
 #  - ~/.aws/credentials in shell-export form (export AWS_ACCESS_KEY_ID=...):
-#    source it as env vars and point CLI file paths at /dev/null so the CLI
-#    uses env vars instead of trying (and failing) to parse it as INI
-#  - otherwise normal AWS resolution via AWS_PROFILE (eg SSO:
-#    aws sso login --profile=<p> && export AWS_PROFILE=<p>)
-if [ -f ~/.aws/credentials ]; then
+#    load environment variables and hide non-INI file from CLI
+_bench_ec2_dir="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
+if [ ! -f "$_bench_ec2_dir/aws.local.env" ]; then
+  echo "missing $_bench_ec2_dir/aws.local.env" >&2
+  echo "  set BENCH_AWS_ACCOUNT and BENCH_AWS_PROFILE in that file" >&2
+  unset _bench_ec2_dir
+  return 1
+fi
+# shellcheck disable=SC1091
+source "$_bench_ec2_dir/aws.local.env"
+unset _bench_ec2_dir
+
+: "${BENCH_AWS_ACCOUNT:?set BENCH_AWS_ACCOUNT in bench/ec2/aws.local.env}"
+export BENCH_AWS_ACCOUNT
+# Pass expected account to Terraform
+export TF_VAR_account_id="$BENCH_AWS_ACCOUNT"
+
+# Detect INI-formatted credentials
+bench_aws_ini_credentials() { grep -qE '^[[:space:]]*\[' ~/.aws/credentials 2>/dev/null; }
+
+: "${AWS_PROFILE:=${BENCH_AWS_PROFILE:-}}"
+if [ -n "$AWS_PROFILE" ]; then
+  export AWS_PROFILE
+  bench_aws_ini_credentials || export AWS_SHARED_CREDENTIALS_FILE=/dev/null
+elif [ -f ~/.aws/credentials ] && ! bench_aws_ini_credentials; then
   set -a
   # shellcheck disable=SC1090
   source ~/.aws/credentials
   set +a
-  unset AWS_PROFILE
   export AWS_SHARED_CREDENTIALS_FILE=/dev/null
   export AWS_CONFIG_FILE=/dev/null
-  # config file is disabled above, so a region must come from env
-  export AWS_DEFAULT_REGION="${AWS_DEFAULT_REGION:-ap-south-1}"
 else
-  : "${AWS_PROFILE:?no ~/.aws/credentials; export AWS_PROFILE (after aws sso login --profile=<p>)}"
+  echo "no profile for AWS account $BENCH_AWS_ACCOUNT" >&2
+  echo "  echo BENCH_AWS_PROFILE=<p> >> bench/ec2/aws.local.env && aws sso login --profile=<p>" >&2
+  return 1
 fi
+# Keep region available when config file is disabled
+export AWS_DEFAULT_REGION="${AWS_DEFAULT_REGION:-ap-south-1}"
+
+# Stop before changing an unexpected account
+bench_aws_account_check() {
+  local got hint="${AWS_PROFILE:+ --profile=$AWS_PROFILE}"
+  got="$(aws sts get-caller-identity --query Account --output text 2>&1)" || {
+    echo "aws sts get-caller-identity failed: $got" >&2
+    echo "  expired SSO session? aws sso login$hint" >&2
+    return 1
+  }
+  [ "$got" = "$BENCH_AWS_ACCOUNT" ] || {
+    echo "wrong AWS account: creds${AWS_PROFILE:+ (profile $AWS_PROFILE)} are in $got, bench infra is in $BENCH_AWS_ACCOUNT" >&2
+    echo "  set BENCH_AWS_PROFILE in bench/ec2/aws.local.env to a profile for $BENCH_AWS_ACCOUNT," >&2
+    echo "  or set BENCH_AWS_ACCOUNT to work in $got" >&2
+    return 1
+  }
+  export AWS_ACCOUNT_ID="$got"
+}
+bench_aws_account_check
+
+# Pass checked credentials to Terraform
+bench_aws_export_creds() {
+  local env_out
+  env_out="$(aws configure export-credentials --format env-no-export 2>/dev/null)" || return 0
+  set -a
+  eval "$env_out"
+  set +a
+  unset AWS_PROFILE
+}
+bench_aws_export_creds

bench/ec2/ec2-bench/deploy.sh

@@ -1,40 +1,41 @@
 #!/usr/bin/env bash
 # Build the bench image, ship it + the current state.env files to the runner
 # box, and install a `walshadow-ec2-bench` wrapper that runs the bench in a
-# host-network container. Then benches run IN the VPC (private IPs, no WAN RTT).
+# host-network container, with the on-box paths already filled in. Benches then
+# run IN the VPC (private IPs, no WAN round trip in the numbers).
 #
-# After this: ssh to the box and run, e.g.
-#   walshadow-ec2-bench --network private --dest clickhouse --bench single-row --state-dir /opt/bench/ec2
-#   # or all four via the shipped runner:
-#   BIN=walshadow-ec2-bench STATE_DIR=/opt/bench/ec2 NETWORK=private DEST=clickhouse \
-#     SKIP_BUILD=1 /opt/bench/run_bench_suite.sh myrun
+# `../stack.sh bench run <name>` drives this end to end. By hand, on the box:
+#   walshadow-ec2-bench --bench single-row
+#   walshadow-ec2-bench --suite myrun          # all four shapes
 set -euo pipefail
 cd "$(dirname "$0")"
 source ./state.env   # PUBLIC_IP, PEM, ...
 source ../lib.sh
 
 IMAGE="${IMAGE:-walshadow-bench:local}"
-REPO_ROOT="$(cd ../../.. && pwd)"
 node_ssh_setup
 
 echo "building $IMAGE (from docker/Dockerfile.bench)…"
-docker build -f "$REPO_ROOT/docker/Dockerfile.bench" -t "$IMAGE" "$REPO_ROOT"
+docker build -f "$(repo_root)/docker/Dockerfile.bench" -t "$IMAGE" "$(repo_root)"
 
 wait_cloud_init
 
-if [ "${FORCE:-0}" != "1" ] && "${SSH[@]}" "sudo docker image inspect $IMAGE >/dev/null 2>&1"; then
-  echo "image $IMAGE already on host (FORCE=1 to resend)"
+REMOTE_IMAGE="$(remote_image_tag "$IMAGE")"
+if [ "${FORCE:-0}" != "1" ] && [ -n "$REMOTE_IMAGE" ]; then
+  echo "image $REMOTE_IMAGE already on host (FORCE=1 to resend)"
 else
   echo "shipping $IMAGE (docker save | ssh | docker load)…"
   docker save "$IMAGE" | gzip | "${SSH[@]}" 'gunzip | sudo docker load'
+  REMOTE_IMAGE="$(remote_image_tag "$IMAGE")"
+  [ -n "$REMOTE_IMAGE" ] || { echo "$IMAGE missing on host after load" >&2; exit 1; }
 fi
 
 # Ship the sibling state.env files so --network private can resolve endpoints.
 echo "shipping endpoint state.env files…"
 # List every level explicitly so /opt/bench and /opt/bench/ec2 are also
-# ubuntu-owned (install -d only reliably applies -o to the leaf dirs) — needed
-# so we can scp run_bench_suite.sh there and the runner can write results.
-"${SSH[@]}" 'sudo install -d -o ubuntu /opt/bench /opt/bench/ec2 /opt/bench/ec2/ec2-source-pg /opt/bench/ec2/ec2-clickhouse /opt/bench/ec2/ec2-pg-standby'
+# ubuntu-owned (install -d only reliably applies -o to the leaf dirs) — the scp
+# below writes as ubuntu, and results stay readable without sudo.
+"${SSH[@]}" 'sudo install -d -o ubuntu /opt/bench /opt/bench/results /opt/bench/ec2 /opt/bench/ec2/ec2-source-pg /opt/bench/ec2/ec2-clickhouse /opt/bench/ec2/ec2-pg-standby'
 for n in ec2-source-pg ec2-clickhouse ec2-pg-standby; do
   if [ -f "../$n/state.env" ]; then
     "${SCP[@]}" "../$n/state.env" "ubuntu@$PUBLIC_IP:/opt/bench/ec2/$n/state.env"
@@ -43,22 +44,20 @@ for n in ec2-source-pg ec2-clickhouse ec2-pg-standby; do
 done
 
 # Install a wrapper: `walshadow-ec2-bench …` → runs the image with host
-# networking and /opt/bench/ec2 mounted at the same path (so --state-dir works).
+# networking and /opt/bench mounted at the same path, carrying the on-box
+# state.env and results locations. Later flags override these (clap keeps the
+# last occurrence), so `--network public` or another dir still works.
 echo "installing walshadow-ec2-bench wrapper…"
 "${SSH[@]}" "cat | sudo tee /usr/local/bin/walshadow-ec2-bench >/dev/null && sudo chmod +x /usr/local/bin/walshadow-ec2-bench" <<WRAP
 #!/usr/bin/env bash
-exec sudo docker run --rm --network host -v /opt/bench/ec2:/opt/bench/ec2 $IMAGE "\$@"
+exec sudo docker run --rm --network host -v /opt/bench:/opt/bench $REMOTE_IMAGE \\
+  --state-dir /opt/bench/ec2 --results-dir /opt/bench/results "\$@"
 WRAP
 
-# Ship run_bench_suite.sh for the all-four pass (BIN/STATE_DIR overridable).
-"${SCP[@]}" "$REPO_ROOT/bench/run_bench_suite.sh" "ubuntu@$PUBLIC_IP:/opt/bench/run_bench_suite.sh"
-"${SSH[@]}" 'chmod +x /opt/bench/run_bench_suite.sh'
-
 echo
 echo "=== ready ==="
-echo "ssh -i $PEM ubuntu@$PUBLIC_IP"
-echo "then, in-VPC (private IPs):"
-echo "  walshadow-ec2-bench --network private --dest clickhouse --bench single-row --state-dir /opt/bench/ec2"
-echo "  # all four into /opt/bench/results/<name>:"
-echo "  BIN=walshadow-ec2-bench STATE_DIR=/opt/bench/ec2 NETWORK=private DEST=clickhouse SKIP_BUILD=1 /opt/bench/run_bench_suite.sh myrun"
-echo "  # for the pg standby:  DEST=postgres …"
+echo "usual path:  ../stack.sh bench run <name>   # runs here, results land in bench/results/<name>"
+echo "by hand:     ssh -i $PEM ubuntu@$PUBLIC_IP"
+echo "  walshadow-ec2-bench --bench single-row"
+echo "  walshadow-ec2-bench --suite myrun         # all four shapes"
+echo "  # for the pg standby:  --dest postgres …"