DD: config-driven bounded-time rollback of shard-encoded location metadata by saintstack · Pull Request #13752 · apple/foundationdb

saintstack · 2026-07-20T17:56:28Z

(Rewrite of the description after a request to move this PR to be totally configuration based which makes the PR better but meant more change. Intent is to backport this to 7.4.)

Rolling SHARD_ENCODE_LOCATION_METADATA true->false could not reach
audit_storage metadata_encoding ROLLBACK COMPLETE in bounded time:
converged and "safe to downgrade binary" was unverifiable short of a
storage wiggle (hours-to-days). This adds an opt-in, bounded-time DD-init
rewrite that converges existing keyServers AND serverKeys back to old
format and clears DataMoveMetaData, driven by two new DatabaseConfiguration
options:

configure shard_metadata_format = original | encoded (target format)
configure shard_metadata_migration = enabled | disabled (run rewrite)

Config is the source of truth for DD's encoding target, resolved once at
init as shard_metadata_format ?? SHARD_ENCODE_LOCATION_METADATA (the knob
is only the fallback when the config is UNSET) and used across all DD
write/move paths -- so a rollback is driven by configure shard_metadata_format=original with no knob flip and no process restart
(the configure triggers a recovery that re-inits DD). Deploying the binary
is a byte-identical no-op until an operator opts in. A completion sentinel
records ROLLBACK COMPLETE and fast-paths later inits. serverKeys are
rewritten serially per storage server, so rewrite time scales with cluster
size (minutes typically); progress is visible via DDShardEncode* trace
events and the audit.

Caveats (details in design/shard-encode-location-metadata.md and
documentation/sphinx/source/command-line-interface.rst):

Binary downgrade: a pre-config binary reads only the knob, so downgrade
is safe only at ROLLBACK COMPLETE with the target binary deployed
SHARD_ENCODE_LOCATION_METADATA=false.
Known limitation: large teams (ddLargeTeamEnabled) and physical shard
(ENABLE_DD_PHYSICAL_SHARD), mutually exclusive with shard encoding, still
read the raw knob, not the config. Once shard_metadata_format is set,
don't move the knob except for a downgrade.

Tested: sim tests/fast/ShardEncodeRollback.toml (knob-fallback path) and
tests/fast/ShardEncodeRollbackConfig.toml (config-only, no knob flip), each
passing a 100,000-run joshua ensemble; plus a live-cluster
forward->rollback->re-forward under mako load with DD-pod kills
(test_shardencode_rollover_load, fdb-kubernetes-tests).

Builds on #13310 (SHARD_ENCODE_LOCATION_METADATA audit + rollback support).
Depends on #13670 (audit_storage counting fix, merged) to observe ROLLBACK
COMPLETE.

Here are two test runs. The first is just exercising the new test. The second is all joshua tests:

  20260724-183324-stack-pr2-shardencode-rollb-24db56d8fdb29f9d compressed=True data_size=38355992 duration=16719145 ended=100000 fail_fast=10 max_runs=100000 pass=100000 priority=100 remaining=0 runtime=1:19:43 sanity=False started=100000 stopped=20260724-195307 submitted=20260724-183324 timeout=5400 username=stack-pr2-shardencode-rollback-rewrite-single
  20260724-183332-stack-pr2-shardencode-rollb-1ff9ed3b7402c6ee compressed=True data_size=38345170 duration=3530901 ended=100000 fail_fast=10 max_runs=100000 pass=100000 priority=100 remaining=0 runtime=0:35:21 sanity=False started=100000 stopped=20260724-190853 submitted=20260724-183332 timeout=5400 username=stack-pr2-shardencode-rollback-rewrite-all

gxglass

Initial review attached. Haven't looked closely here, probably should since there is a lot going on

pr-13752-review.md

bnamasivayam · 2026-07-21T20:51:31Z

+shard-location metadata (``\xff/keyServers/`` and ``\xff/serverKeys/``)
+between the legacy tag-based encoding and the newer UID+dataMoveId
+encoding introduced with ``SHARD_ENCODE_LOCATION_METADATA``. The pair
+mirrors the ``storage_engine`` + ``perpetual_storage_wiggle`` pattern.


I think the mirrored pattern is storage_engine + storage_migration_type

bnamasivayam · 2026-07-21T20:52:42Z

 ---------

-The ``configure`` command changes the database configuration. Its syntax is ``configure [new|tss] [single|double|triple|three_data_hall|three_datacenter] [ssd|memory] [grv_proxies=<N>] [commit_proxies=<N>] [resolvers=<N>] [logs=<N>] [count=<TSS_COUNT>] [perpetual_storage_wiggle=<WIGGLE_SPEED>] [perpetual_storage_wiggle_locality=<<LOCALITY_KEY>:<LOCALITY_VALUE>|0>] [perpetual_storage_wiggle_engine=<ENGINE>] [storage_migration_type={disabled|aggressive|gradual}]``.
+The ``configure`` command changes the database configuration. Its syntax is ``configure [new|tss] [single|double|triple|three_data_hall|three_datacenter] [ssd|memory] [grv_proxies=<N>] [commit_proxies=<N>] [resolvers=<N>] [logs=<N>] [count=<TSS_COUNT>] [perpetual_storage_wiggle=<WIGGLE_SPEED>] [perpetual_storage_wiggle_locality=<<LOCALITY_KEY>:<LOCALITY_VALUE>|0>] [perpetual_storage_wiggle_engine=<ENGINE>] [storage_migration_type={disabled|aggressive|gradual}] [shard_metadata_format={old_format|new_format}] [shard_metadata_migration={enabled|disabled}]``.


I think the format should have a more qualified name like {tag-based-format | uuid-based-format}

bnamasivayam · 2026-07-21T22:06:20Z

+   the next DD does a full scan and converges.
+
+The rollback procedure (with active rewrite):
+1. Set `SHARD_ENCODE_LOCATION_METADATA=false` and restart the


Keeping this knob around makes this logic brittle. Knob rollouts are not synchronous and different processes can have different values for knobs during the rollout. I think this feature is better to be controlled only by config variables.

Nod. In this case I think we'd be fine since its just the DD that is the reader.

Offline, you argued we should leave the knob behind and move it all over to a configuration instead. The commit message has my reasoning on why I avoided making this transition in a point release (fatter patch with changes in core read/write path) but let me put up a patch that is pure configuration. No harm.

Thanks for the review B

saintstack · 2026-07-21T22:41:26Z

pr-13752-review.md

@gxglass Does the link work for you (404 for me)? Maybe wait and regen after I put up next patch because a bunch of change after chat w/ Bala #13752 (comment) (Thanks for review)

…adata Rolling SHARD_ENCODE_LOCATION_METADATA true->false could not reach `audit_storage metadata_encoding` ROLLBACK COMPLETE in bounded time: converged and "safe to downgrade binary" was unverifiable short of a storage wiggle (hours-to-days). This adds an opt-in, bounded-time DD-init rewrite that converges existing keyServers AND serverKeys back to old format and clears DataMoveMetaData, driven by two new DatabaseConfiguration options: configure shard_metadata_format = original | encoded (target format) configure shard_metadata_migration = enabled | disabled (run rewrite) Config is the source of truth for DD's encoding target, resolved once at init as `shard_metadata_format ?? SHARD_ENCODE_LOCATION_METADATA` (the knob is only the fallback when the config is UNSET) and used across all DD write/move paths -- so a rollback is driven by `configure shard_metadata_format=original` with no knob flip and no process restart (the configure triggers a recovery that re-inits DD). Deploying the binary is a byte-identical no-op until an operator opts in. A completion sentinel records ROLLBACK COMPLETE and fast-paths later inits. serverKeys are rewritten serially per storage server, so rewrite time scales with cluster size (minutes typically); progress is visible via DDShardEncode* trace events and the audit. Caveats (details in design/shard-encode-location-metadata.md and documentation/sphinx/source/command-line-interface.rst): - Binary downgrade: a pre-config binary reads only the knob, so downgrade is safe only at ROLLBACK COMPLETE with the target binary deployed SHARD_ENCODE_LOCATION_METADATA=false. - Known limitation: large teams (ddLargeTeamEnabled) and physical shard (ENABLE_DD_PHYSICAL_SHARD), mutually exclusive with shard encoding, still read the raw knob, not the config. Once shard_metadata_format is set, don't move the knob except for a downgrade. Tested: sim tests/fast/ShardEncodeRollback.toml (knob-fallback path) and tests/fast/ShardEncodeRollbackConfig.toml (config-only, no knob flip), each passing a 100,000-run joshua ensemble; plus a live-cluster forward->rollback->re-forward under mako load with DD-pod kills (test_shardencode_rollover_load, fdb-kubernetes-tests). Builds on apple#13310 (SHARD_ENCODE_LOCATION_METADATA audit + rollback support). Depends on apple#13670 (audit_storage counting fix, merged) to observe ROLLBACK COMPLETE.

saintstack · 2026-08-14T17:02:37Z

Rebase. Reran joshua: 20260814-162701-stack_rebase-d082f3fbaf691bb5 compressed=True data_size=38650374 duration=3696370 ended=100000 fail_fast=10 max_runs=100000 pass=100000 priority=100 remaining=0 runtime=0:32:49 sanity=False started=100000 stopped=20260814-165950 submitted=20260814-162701 timeout=5400 username=stack_rebase

bnamasivayam

The user facing commands part of this PR looks good to me.

foundationdb-ci · 2026-08-14T17:49:05Z

Result of foundationdb-pr-clang-arm on Linux RHEL 9

Commit ID: 3c1a0ef
Duration 0:46:47
Result: ✅ SUCCEEDED
Error: N/A
Build Log terminal output (available for 30 days)
Build Workspace zip file of the working directory (available for 30 days)

foundationdb-ci · 2026-08-14T17:55:10Z

Result of foundationdb-pr-clang-ide on Linux RHEL 9

Commit ID: 3c1a0ef
Duration 0:52:53
Result: ✅ SUCCEEDED
Error: N/A
Build Log terminal output (available for 30 days)
Build Workspace zip file of the working directory (available for 30 days)

foundationdb-ci · 2026-08-14T18:17:38Z

Result of foundationdb-pr on Linux RHEL 9

Commit ID: 3c1a0ef
Duration 1:15:19
Result: ✅ SUCCEEDED
Error: N/A
Build Log terminal output (available for 30 days)
Build Workspace zip file of the working directory (available for 30 days)

foundationdb-ci · 2026-08-14T18:28:58Z

Result of foundationdb-pr-clang on Linux RHEL 9

Commit ID: 3c1a0ef
Duration 1:26:41
Result: ✅ SUCCEEDED
Error: N/A
Build Log terminal output (available for 30 days)
Build Workspace zip file of the working directory (available for 30 days)

foundationdb-ci · 2026-08-14T18:33:42Z

Result of foundationdb-pr-cluster-tests on Linux RHEL 9

Commit ID: 3c1a0ef
Duration 1:31:24
Result: ✅ SUCCEEDED
Error: N/A
Build Log terminal output (available for 30 days)
Build Workspace zip file of the working directory (available for 30 days)
Cluster Test Logs zip file of the test logs (available for 30 days)

foundationdb-ci · 2026-08-14T19:21:02Z

Result of foundationdb-pr-macos-m1 on macOS 14.x

Commit ID: 3c1a0ef
Duration 2:18:42
Result: ✅ SUCCEEDED
Error: N/A
Build Log terminal output (available for 30 days)
Build Workspace zip file of the working directory (available for 30 days)

foundationdb-ci · 2026-08-14T21:56:58Z

Result of foundationdb-pr-macos on macOS 14.x

Commit ID: 3c1a0ef
Duration 4:54:36
Result: ✅ SUCCEEDED
Error: N/A
Build Log terminal output (available for 30 days)
Build Workspace zip file of the working directory (available for 30 days)

saintstack requested review from bnamasivayam, neethuhaneesha and spraza July 20, 2026 17:56

This comment has been minimized.

Sign in to view

gxglass reviewed Jul 20, 2026

View reviewed changes

bnamasivayam reviewed Jul 21, 2026

View reviewed changes

bnamasivayam requested changes Jul 21, 2026

View reviewed changes

saintstack force-pushed the rollback branch from 4b3a483 to df55891 Compare July 25, 2026 00:20

saintstack changed the title ~~DD: bounded-time rollback of shard-encoded location metadata~~ DD: config-driven bounded-time rollback of shard-encoded location metadata Jul 25, 2026

saintstack requested review from bnamasivayam and gxglass July 25, 2026 00:25

This comment has been minimized.

Sign in to view

michael stack added 2 commits August 14, 2026 09:01

Formatting

3c1a0ef

saintstack force-pushed the rollback branch from a67522b to 3c1a0ef Compare August 14, 2026 17:02

bnamasivayam reviewed Aug 14, 2026

View reviewed changes

saintstack requested a review from bnamasivayam August 14, 2026 17:37

Uh oh!

Conversation

saintstack commented Jul 20, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

gxglass left a comment

Choose a reason for hiding this comment

Uh oh!

bnamasivayam Jul 21, 2026

Choose a reason for hiding this comment

Uh oh!

bnamasivayam Jul 21, 2026

Choose a reason for hiding this comment

Uh oh!

bnamasivayam Jul 21, 2026

Choose a reason for hiding this comment

Uh oh!

saintstack Jul 21, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

saintstack commented Jul 21, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

saintstack commented Aug 14, 2026

Uh oh!

bnamasivayam left a comment

Choose a reason for hiding this comment

Uh oh!

foundationdb-ci commented Aug 14, 2026

Result of foundationdb-pr-clang-arm on Linux RHEL 9

Uh oh!

foundationdb-ci commented Aug 14, 2026

Result of foundationdb-pr-clang-ide on Linux RHEL 9

Uh oh!

foundationdb-ci commented Aug 14, 2026

Result of foundationdb-pr on Linux RHEL 9

Uh oh!

foundationdb-ci commented Aug 14, 2026

Result of foundationdb-pr-clang on Linux RHEL 9

Uh oh!

foundationdb-ci commented Aug 14, 2026

Result of foundationdb-pr-cluster-tests on Linux RHEL 9

Uh oh!

foundationdb-ci commented Aug 14, 2026

Result of foundationdb-pr-macos-m1 on macOS 14.x

Uh oh!

foundationdb-ci commented Aug 14, 2026

Result of foundationdb-pr-macos on macOS 14.x

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

saintstack commented Jul 20, 2026 •

edited

Loading

saintstack Jul 21, 2026 •

edited

Loading

saintstack commented Jul 21, 2026 •

edited

Loading