feat(cli): cache the shadow database baseline as a PGDATA snapshot (CLI-2191) - #6184
Open
avallete wants to merge 91 commits into
Open
feat(cli): cache the shadow database baseline as a PGDATA snapshot (CLI-2191)#6184avallete wants to merge 91 commits into
avallete wants to merge 91 commits into
Conversation
…lta-next # Conflicts: # apps/cli-go/cmd/db.go # apps/cli/docs/go-cli-porting-status.md # apps/cli/src/legacy/commands/db/diff/SIDE_EFFECTS.md # apps/cli/src/legacy/commands/db/diff/diff.handler.ts # apps/cli/src/legacy/commands/db/diff/diff.integration.test.ts # apps/cli/src/legacy/commands/db/diff/diff.layers.ts # apps/cli/src/legacy/commands/db/pull/SIDE_EFFECTS.md # apps/cli/src/legacy/commands/db/pull/pull.handler.ts # apps/cli/src/legacy/commands/db/pull/pull.integration.test.ts # apps/cli/src/legacy/commands/db/pull/pull.layers.ts # apps/cli/src/legacy/commands/db/push/SIDE_EFFECTS.md # apps/cli/src/legacy/commands/db/reset/SIDE_EFFECTS.md # apps/cli/src/legacy/commands/db/schema/declarative/declarative.orchestrate.integration.test.ts # apps/cli/src/legacy/commands/db/schema/declarative/declarative.orchestrate.ts # apps/cli/src/legacy/commands/db/schema/declarative/generate/generate.handler.ts # apps/cli/src/legacy/commands/db/schema/declarative/generate/generate.integration.test.ts # apps/cli/src/legacy/commands/db/schema/declarative/sync/SIDE_EFFECTS.md # apps/cli/src/legacy/commands/db/schema/declarative/sync/sync.handler.ts # apps/cli/src/legacy/commands/db/schema/declarative/sync/sync.integration.test.ts # apps/cli/src/legacy/commands/db/shared/legacy-pgdelta.seam.layer.ts # apps/cli/src/legacy/commands/db/shared/legacy-pgdelta.seam.service.ts # apps/cli/src/legacy/shared/db-bootstrap/db-setup.ts # apps/cli/src/legacy/shared/legacy-db-connection.sql-pg.layer.ts # apps/cli/src/legacy/shared/legacy-http-errors.ts # apps/cli/src/legacy/shared/legacy-migration-apply.ts # apps/cli/src/legacy/shared/legacy-migration-apply.unit.test.ts # apps/cli/src/legacy/shared/legacy-pgdelta.cache.ts # packages/api/src/effect.ts # packages/api/src/internal/client.ts # packages/api/src/internal/client.unit.test.ts
…lta-next # Conflicts: # apps/cli/src/legacy/commands/db/diff/SIDE_EFFECTS.md # apps/cli/src/legacy/commands/db/diff/diff.handler.ts # apps/cli/src/legacy/commands/db/diff/diff.integration.test.ts # apps/cli/src/legacy/commands/db/pull/SIDE_EFFECTS.md # apps/cli/src/legacy/commands/db/shared/legacy-shadow-source.ts
Legacy migrate, next migrate, and next declarative bake different pg_net states into the same cluster recipe. Hash that policy so they cannot share a snapshot, honor --no-cache on the remaining catalog exports, and document the global cache's LRU/TTL side effects.
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 7a8fa09b16
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
# Conflicts: # apps/cli/src/legacy/commands/db/diff/SIDE_EFFECTS.md
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 788e9186d7
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 09d8c6ccc9
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
4 tasks
avallete
pushed a commit
that referenced
this pull request
Aug 16, 2026
The stale-peek refresh comment listed the remote JWKS among the inputs a delayed acquire re-resolves, but that resolver is deliberately memoized per shadow input (shadow-database.ts, review: Codex on #6184) so the cache key and the baked baseline always carry the same value and cannot diverge — a delayed acquire keeps the command-start JWKS, well inside the staleness the snapshot cache accepts by design (a warm hit serves a tar up to 14 days old under its matching key). Scope the comment to roles.sql, the one input the refresh actually re-reads, and state the JWKS exemption explicitly. Review: Codex on #6215. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01NYVv1jjXdrTzqDTVrU4Had
This was referenced Aug 16, 2026
…cli-pr-conflicts-529506 # Conflicts: # apps/cli/docs/go-cli-divergences.md # apps/cli/package.json # apps/cli/src/legacy/commands/db/diff/SIDE_EFFECTS.md # apps/cli/src/legacy/commands/db/diff/diff.handler.ts # apps/cli/src/legacy/commands/db/diff/diff.integration.test.ts # apps/cli/src/legacy/commands/db/pull/SIDE_EFFECTS.md # apps/cli/src/legacy/commands/db/pull/pull.handler.ts # apps/cli/src/legacy/commands/db/pull/pull.integration.test.ts # apps/cli/src/legacy/commands/db/schema/declarative/declarative.flow.ts # apps/cli/src/legacy/commands/db/schema/declarative/declarative.flow.unit.test.ts # apps/cli/src/legacy/commands/db/schema/declarative/declarative.orchestrate.ts # apps/cli/src/legacy/commands/db/schema/declarative/generate/SIDE_EFFECTS.md # apps/cli/src/legacy/commands/db/schema/declarative/generate/generate.command.ts # apps/cli/src/legacy/commands/db/schema/declarative/generate/generate.handler.ts # apps/cli/src/legacy/commands/db/schema/declarative/generate/generate.integration.test.ts # apps/cli/src/legacy/commands/db/schema/declarative/sync/SIDE_EFFECTS.md # apps/cli/src/legacy/commands/db/schema/declarative/sync/sync.handler.ts # apps/cli/src/legacy/commands/db/schema/declarative/sync/sync.integration.test.ts # apps/cli/src/legacy/commands/db/shared/legacy-pgdelta-engine.layer.ts # apps/cli/src/legacy/commands/db/shared/legacy-pgdelta-engine.layer.unit.test.ts # apps/cli/src/legacy/commands/db/shared/legacy-pgdelta-engine.next.layer.integration.test.ts # apps/cli/src/legacy/commands/db/shared/legacy-pgdelta-engine.next.layer.ts # apps/cli/src/legacy/commands/db/shared/legacy-pgdelta-engine.next.layer.unit.test.ts # apps/cli/src/legacy/commands/db/shared/legacy-pgdelta-next-adapter.layer.ts # apps/cli/src/legacy/commands/db/shared/legacy-pgdelta-next-adapter.unit.test.ts # apps/cli/src/legacy/commands/db/shared/legacy-pgdelta-next-artifacts.ts # apps/cli/src/legacy/commands/db/shared/legacy-pgdelta-next-artifacts.unit.test.ts # apps/cli/src/legacy/commands/db/shared/legacy-pgdelta-next-shadow.layer.ts # apps/cli/src/legacy/commands/db/shared/legacy-pgdelta-next-shadow.layer.unit.test.ts # apps/cli/src/legacy/commands/db/shared/legacy-pgdelta-next-shadow.service.ts # apps/cli/src/legacy/commands/db/shared/legacy-pgdelta.write.unit.test.ts # apps/cli/src/legacy/commands/db/shared/legacy-shadow-source.ts # apps/cli/src/legacy/shared/db-bootstrap/db-setup.ts # apps/cli/src/legacy/shared/db-bootstrap/db-setup.unit.test.ts # apps/cli/src/legacy/shared/db-bootstrap/shadow-database.ts # apps/cli/src/legacy/shared/legacy-db-push-core.ts # apps/cli/src/legacy/shared/legacy-pgdelta-next-flag.ts # apps/cli/src/legacy/shared/legacy-pgdelta-next-flag.unit.test.ts # apps/cli/tests/fixtures/compiled-libpg-query.ts # pnpm-lock.yaml # pnpm-workspace.yaml
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
- Bypass pg-delta's same-identity guard by snapshot lineage (same cache key), not by requiring both plan shadows to be warm restores — the first cold plan exports the tar its declarative sibling restores, and the guard rejected that clone (Codex P1). - Fold LEGACY_START_ENABLE_DATABASE_WEBHOOKS_SQL into the baseline SQL digest so editing it invalidates stale pg_net baselines. - Bound legacyWaitForShadowReady by elapsed time (+ one connect allowance) so hung dials cannot stretch the wait ~3x its budget. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…eview findings Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 0430203364
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
…l run db diff/db pull hardcoded webhooks: "enabled" into the cache key even when pgdelta-next's config-following migrate would run, and db diff's explicit migrations-catalog path omitted the policy entirely while its provisioner forces pg_net on — either mismatch lets the two engines restore each other's tars. Derive the key from migrationMode at the diff/pull seams, declare "enabled" at the explicit catalog call, and drop exportViaShadowCatalog's opts default so callers must state their policy. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Rapid-fire publishes can land within the filesystem's timestamp granularity, making the "oldest" tar ambiguous and the eviction pick arbitrary (CI-only flake). Age the first tar explicitly — the test asserts the keep-cap, not tie-breaking. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
db diff and db pull each gain a cache-enabled scenario proving a legacy-engine baseline tar is never restored into a pg-delta-next run (and vice versa each publishes its own key). Mutation-verified: reverting either call site to a hardcoded webhooks: "enabled" fails the new test at the warm-restore assertion. The shadow-cache suite's stateful Docker model is hoisted into tests/helpers for reuse; the stateless shadow spawner mock stays byte-identical for existing callers. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 9970ce432d
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
…key token - A warm hit now scans the tar's entry headers for the cluster marker (data/PG_VERSION) before restoring: an empty or foreign-but-valid tar no longer initdbs a bare cluster that reports baselinePresent and skips setup — it is discarded as suspect and the run cold-provisions. - The cache key hashes the EFFECTIVE api.auto_expose_new_tables behavior (grants kept vs revoked): unset and explicit false execute identical revoke SQL, so they no longer force separate ~90MB tars. - Docs: generate joins the cache divergence entry; roles.sql cache-key reads listed across diff/pull/sync/generate SIDE_EFFECTS. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 9baed89bbc
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
The export now writes data/SUPABASE_BASELINE into the stopped shadow's PGDATA right before the outbound docker cp, and warm-hit validation requires it alongside data/PG_VERSION — a valid bare-cluster tar is discarded as suspect instead of restoring into baselinePresent: true and skipping setup. Only snapshotBaseline stamps, strictly after the platform baseline, so a snapshot taken too early can never carry the marker (regression guard). Uppercase name keeps the entry at the front of docker cp's sorted tar so warm validations stay a first-blocks read. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 4d494f4d96
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
…tants The baseline marker's content is now the snapshot's own cache key and warm validation verifies it, so a valid snapshot copied over another key's filename is discarded (wrong-key verdict, distinct from a missing marker) instead of restoring a mismatched baseline. The baseline digest (renamed baseline_embedded_digest) now also hashes the Realtime seed constants persisted by the one-shot job (tenant id, encryption key, db user/name/port), so editing them without an image bump invalidates stale tars. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: b382f94904
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
db diff/db pull/ declarative sync rebuild the shadow's platform baseline (initdb + init SQL + the PG15+ one-shot realtime/storage/auth jobs, ~15s) on every run, before user migrations even start. This PR makes that a once-per-config cost. Measured on a default-services PG17 project: ~3.3s warm provision vs ~15.5s cold.What changed
fix): the shadow's docker healthcheck can't report before t+10s (10s interval, no start_period), solegacyWaitForShadowReadynow polls container-state + a short authenticated connect instead. Same timeout budget, same error shape and log dump; the healthcheck config and the long-runningdbcontainer's wait are untouched.exec'd postgres entrypoint (fix): PID 1 wassh, which swallowed SIGTERM, so everydocker stopburned the 10s grace period. All three entrypoint variants nowexec, and stops take ~1s (also benefitssupabase stop). Deliberate divergence from Go's script; timing is outside the Go-parity surface (ADR 0016).feat, default ON): right after the baseline and before any user migration, the cold path stops the shadow and streams its PGDATA tosupabase/.temp/pgdelta/shadow-baseline-<key>.tar(~90MB, atomic rename, current key only). A key-matching run unpacks the tar into a fresh container beforedocker start— the entrypoint skips initdb, the baseline is skipped, migrations apply directly.Design notes for review
db-bootstrap/pgdata-snapshot.ts(container-agnostic, plain-file artifact) — reusable for a future live-stack savepoint or native (non-Docker) Postgres; a TODO there records the hot-save modes (docker pausefreeze-copy,pg_backup_start/stoponline copy).docker cpis load-bearing: the directory form resets ownership to root and Postgres refuses the data dir.--rm, because Docker destroys an AutoRemove container ondocker stop(verified) — release still removes every shadow; a SIGKILLed CLI leaves at most one stopped project-labeled container, swept bysupabase stop.SUPABASE_SHADOW_CACHE=false. Debug timings:SUPABASE_SHADOW_DEBUG=1.Still cold-provisioning (documented follow-ups):
db diff --use-pgadmin,db pull --declarative's raw shadow,migration squash.🤖 Generated with Claude Code