Add AuditType::RangeDigest: a storage-server-side content fingerprint

[saintstack]

Adds AuditType::RangeDigest, a 256-bit content fingerprint of the key-value multiset computed
in parallel by the storage servers that already hold the data. A root taken before a backup equals
the root taken after a restore iff the restored data is identical as a set — regardless of how
the two clusters shard the keyspace.

This gives backup/restore validation a check that is both rigorous (every key-value, no sampling)
and affordable at multi-terabyte scale: only 32-byte digests cross the network, never user data, so
throughput scales with the number of storage servers rather than one client's bandwidth. Useful
testing restore is same as original state.

Design doc: design/range-digest.md (included in this PR).

Motivation

A backup of a multi-terabyte database is only trustworthy if the restore can be confirmed to
reproduce it. The obvious check — a client reads the whole keyspace and hashes it — was built first
as an external prototype (fdbfingerprint) and produced matching roots at 1B and 3B. It was
abandoned as the mechanism on measured cost: the 3B dataset took 9h11m at ~79 MB/s
aggregate, with all of that read load landing on the very path under validation, and a
before/after comparison needs two such runs.

Two qualifications, since the numbers invite an overclaim. That 79 MB/s was not a hard client
ceiling — it was ~4 MB/s across 20 disks that sat ~98% idle, i.e. an under-parallelized prototype
(8 pods, each a single-network-thread client unable to hide cold-read latency); a tuned client would
be considerably faster. And while the server-side digest measures ~2770 MB/s at 10B, that is against
the untuned client, so the ~35× gap is not like-for-like.

The architectural argument stands without the ratio: an external fingerprint scales with whatever
bandwidth one client pool is given and spends read capacity production traffic needs, while hashing
on the servers that already hold the data scales with storage-server count and moves only 32-byte
digests.

The check must also be partition-independent: a restore lands the same logical data under
completely different shard boundaries, so any fingerprint that depends on the physical partition
(Merkle tree over a shard list, per-shard hash comparison) cannot be compared across a restore at
all. Hence an additive multiset hash.

Design

leaf  = SHA-256( u32be len(key) | key | u32be len(value) | value )
digest = ( Σ leaf ) mod 2^256          # 256-bit big-endian accumulator, 0 = empty set
root   = combine of every storage server's per-range digest

Addition mod 2²⁵⁶ is associative and commutative, so the root is independent of grouping, ordering
and shard layout — the partition-independence requirement — and per-range digests combine upward
with no boundary replay. 256 bits matches the SHA-256 leaf (no truncation) and leaves a ~2¹²⁸
accidental-collision margin against datasets of ~2³³ keys, for 32 bytes per range on the wire.

The leaf encoding is byte-for-byte identical to the Phase-0 fdbfingerprint tool, which is retained
as an independent cross-check.

Explicit non-goal: collision resistance against an adversary who chooses inputs. Additive
multiset hashes are vulnerable to subset-sum/lattice attacks; the threat model here is accidental
corruption of FDB's own data on a trusted cluster, so a keyed MAC's key management and cost buy
nothing in scope. This is stated in the design doc rather than left implicit.

Precondition: quiescence

The digest must be taken over a quiescent cluster (writes stopped, moving_data → 0). Two
independent invariants require it:

1. Single version — each server folds its own key-values at its own current read version;
   there is no cluster-wide pinned version, so concurrent writes would hash at inconsistent
   versions and the root would correspond to no single snapshot.
2. Exactly-once — the additive combine assumes each key-value is folded exactly once
   cluster-wide. Under in-flight shard movement a key can be folded by both the losing and gaining
   server, or by neither at the instant it is read.

This is not theoretical: at 100M scale, digesting during data-distribution churn produced a root
mismatch, with both sides counting ~104.8M key-values against a true ~100M. Callers must settle
first — natural for backup/restore, where both sides are settled datasets.

RangeDigest.h carries a TODO for the generalization: folding every server against a single pinned
cluster read version, with an ownership snapshot consistent with that version, would restore both
invariants online. Unnecessary for backup/restore, and it would not change the leaf encoding or the
root of a quiesced dataset, so fingerprints taken today stay comparable.

What's in the PR

area	change
fdbclient/RangeDigest.{h,cpp}	new: leaf encoding, 256-bit accumulator, addKeyValue/combine
fdbserver/storageserver/storageserver.cpp	auditRangeDigestQ, dispatched from serveAuditStorageRequests; batched and rate-limited
fdbserver/datadistributor/DataDistribution.cpp	dispatch, per-range persistence, combine to a cluster root on Complete
fdbclient/AuditUtils.cpp, Audit.h, AuditUtils.h	AuditType::RangeDigest = 7; digest/kvCount/byteCount on the audit record; range-digest summary read
fdbcli/AuditStorageCommand.cpp, GetAuditStatusCommand.cpp	range_digest audit type; new get_audit_status range_digest root <id>
fdbserver/workloads/RangeDigestValidation.cpp, tests/fast/RangeDigestValidation.toml	simulation workload + test registration
design/range-digest.md	design doc

Rate limiting and batching reuse the existing audit controls
(AUDIT_STORAGE_RATE_PER_SERVER_MAX via SpeedLimit, AUDIT_RESTORE_BATCH_KEY_LIMIT,
REPLY_BYTE_LIMIT) rather than introducing new knobs.

CLI surface:

audit_storage range_digest "" \xff          # start; prints the audit ID
get_audit_status range_digest id <id>       # phase, plus KVCount/Bytes/Digest for this type
get_audit_status range_digest progress <id> # per-range detail (while Running)
get_audit_status range_digest root <id>     # combined cluster root (Complete only)

root deliberately refuses to print unless the audit is Complete: per-range progress is cleared
at completion, so a root read earlier could be computed over partial coverage. Relatedly, the
combine is guarded — if the persisted per-range digests do not tile the whole audit range, the audit
retries rather than publishing a root that silently omits keys.

Compatibility

AuditStorageState::serialize gains three appended fields (digest, kvCount, byteCount).
With flatbuffers + IncludeVersion, old readers ignore trailing fields and new readers
default-initialize them for records written by older versions; existing field order is untouched.
The fields are empty/zero for every other audit type.

Otherwise additive and opt-in: a new AuditType invoked on demand, no on-disk format change to user
data, no effect on a cluster that never requests it. Rollback is simply not issuing the audit.

Testing

• Simulation (tests/fast/RangeDigestValidation.toml): the workload writes a known key-value
  set, then (a) cross-checks the audit's combined root against an independent client-side
  computation of the same additive digest — scanning every key-value and applying the canonical
  leaf encoding, which validates the storage-server fold and the combine — and (b) runs a
  second audit and asserts the root is identical, which demonstrates partition-independence
  since data distribution may have moved shards in between. It waits for quiescence before each
  digest. The backup → clear → restore → recompute cycle is covered by the Kubernetes test, not in
  simulation.
• Multi-shard combine exercised on seeds 12345 / 777 / 424242 / 9001 (4–9 shards, up to 18 SS
  digest tasks): 0 SevError, all roots matching the independent computation.
• Scale validation via the companion Kubernetes test (test_backup_restore_rangedigest):
  root_after == root_before verified end-to-end at 100M, 1B, 3B and ~4.3B key-values. A 10B run
  (~9.8e9 KV / ~9.26 TB, root_before = fa1bee7a…) is in flight.
• Measured throughput at 10B: 9,257,887,466,305 bytes over 9,808,347,148 key-values digested in
  53m07s — ~2770 MB/s aggregate — on a 30-machine cluster with 4 storage disks per machine. The
  external Phase-0 client did the smaller 3B dataset in 9h11m at ~79 MB/s, but from an
  under-parallelized client, so treat that as context rather than a tuned comparison.
• Reproducibility: that same 10B dataset re-digested to a byte-identical root on a later run
  against the same cluster, with no restore involved — so a matching root is not an artifact of
  digesting twice in quick succession.

Reviewer notes — known gaps, stated rather than glossed

• A skipped run is also green. If an audit cannot complete under harness churn, the workload
  logs RangeDigestValidationInconclusive and returns without asserting. The content assertions sit
  deliberately outside that try, so a genuine mismatch is never swallowed — but it does mean a
  green result should be confirmed by the presence of RangeDigestValidationSuccess rather than the
  absence of errors. Worth deciding whether inconclusive runs should be made noisier.
• The committed .toml uses a modest dataset (nodeCount=10000) so it stays cheap across the
  Joshua matrix; some seeds will keep the data in a single shard, in which case that seed does
  not exercise the cross-server combine. The multi-shard evidence above comes from a larger
  dedicated run with strictShardCheck=true. Worth deciding whether a strict variant should gate
  here.
• The digest is read-heavy, so its throughput is sensitive to the block-cache-to-data ratio; a
  low-memory configuration makes it disk-bound. An observability note, not a defect.

Companion PR

The Kubernetes scale test that drives this (test_backup_restore_rangedigest in
FoundationDB/fdb-kubernetes-tests).

[foundationdb-ci]

Result of foundationdb-pr-clang-ide on Linux RHEL 9

• Commit ID: f2dfd9a
• Duration 0:26:16
• Result: ✅ SUCCEEDED
• Error: N/A
• Build Log terminal output (available for 30 days)
• Build Workspace zip file of the working directory (available for 30 days)

[saintstack]

20260814-190026-stack_digest-d48b866b93620ce5 compressed=True data_size=38399986 duration=2750044 ended=100000 fail=1 fail_fast=10 max_runs=100000 pass=99999 priority=100 remaining=0 runtime=0:30:11 sanity=False started=100000 stopped=20260814-193037 submitted=20260814-190026 timeout=5400 username=stack_digest

Here is the failure which seems unrelated...

RandomSeed="1982037524" SourceVersion="7da2679aa6d5d1238141fb4de86c7527fe907e8b" Time="1786734739" BuggifyEnabled="1" DeterminismCheck="0" FaultInjectionEnabled="1" TestFile="tests/slow/RyowCorrectness.toml"

[foundationdb-ci]

Result of foundationdb-pr-clang-arm on Linux RHEL 9

• Commit ID: f2dfd9a
• Duration 0:47:51
• Result: ✅ SUCCEEDED
• Error: N/A
• Build Log terminal output (available for 30 days)
• Build Workspace zip file of the working directory (available for 30 days)

[foundationdb-ci]

Result of foundationdb-pr-macos-m1 on macOS 14.x

• Commit ID: f2dfd9a
• Duration 1:00:26
• Result: ✅ SUCCEEDED
• Error: N/A
• Build Log terminal output (available for 30 days)
• Build Workspace zip file of the working directory (available for 30 days)

[foundationdb-ci]

Result of foundationdb-pr-clang on Linux RHEL 9

• Commit ID: f2dfd9a
• Duration 1:10:15
• Result: ❌ FAILED
• Error: Error while executing command: if python3 -m joshua.joshua list --stopped | grep ${ENSEMBLE_ID} | grep -q 'pass=10[0-9][0-9][0-9]'; then echo PASS; else echo FAIL && exit 1; fi. Reason: exit status 1
• Build Log terminal output (available for 30 days)
• Build Workspace zip file of the working directory (available for 30 days)

[foundationdb-ci]

Result of foundationdb-pr on Linux RHEL 9

• Commit ID: f2dfd9a
• Duration 1:11:48
• Result: ✅ SUCCEEDED
• Error: N/A
• Build Log terminal output (available for 30 days)
• Build Workspace zip file of the working directory (available for 30 days)

[foundationdb-ci]

Result of foundationdb-pr-cluster-tests on Linux RHEL 9

• Commit ID: f2dfd9a
• Duration 1:30:07
• Result: ✅ SUCCEEDED
• Error: N/A
• Build Log terminal output (available for 30 days)
• Build Workspace zip file of the working directory (available for 30 days)
• Cluster Test Logs zip file of the test logs (available for 30 days)

[foundationdb-ci]

Result of foundationdb-pr-macos on macOS 14.x

• Commit ID: f2dfd9a
• Duration 4:13:36
• Result: ❌ FAILED
• Error: `Error while executing command: # compatible with bash and zsh
  ssh -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null -i ${HOME}/.ssh_key
  -o ServerAliveInterval=9 ec2-user@${MAC_EC2_HOST} "
  set -e -o noclobber
  BUILDNUM=${CODEBUILD_BUILD_NUMBER}
  WAIT_TIME=14400 # 4 hours
  LOCK_STALE=4500 # 75 minutes
  LOCK=/tmp/ci.lock
  "'

macOS stat flags, not linux compatible

lock_mtime() { stat -f %m $LOCK || echo 0 }
I=0
while ! (echo $BUILDNUM >$LOCK) 2>/dev/null ; do
if (( ++I >= WAIT_TIME )); then
echo "timeout waiting for ci.lock after $I seconds"
exit 1
fi
if (( $(date +%s) - $(lock_mtime) >= LOCK_STALE )); then
echo "lock expired after $LOCK_STALE seconds ..."
# racy but sleep so chances are low
sleep $(( 3 + RANDOM % 8 ))
if (( $(date +%s) - $(lock_mtime) >= LOCK_STALE )); then
echo "lock holder is gone"
rm -f $LOCK
else
echo "new lock holder"
fi
fi
sleep 1
done
echo "acquired ci.lock"
'
. Reason: exit status 1`

• Build Log terminal output (available for 30 days)
• Build Workspace zip file of the working directory (available for 30 days)