Skip to content

Fix old-epoch backup worker cleanup after backup completion - #13800

Merged
tclinkenbeard-oai merged 2 commits into
apple:mainfrom
tclinkenbeard-oai:dev/tclinkenbeard/backup-and-restore-remote-triple-quietdatabase-assert
Aug 13, 2026
Merged

Fix old-epoch backup worker cleanup after backup completion#13800
tclinkenbeard-oai merged 2 commits into
apple:mainfrom
tclinkenbeard-oai:dev/tclinkenbeard/backup-and-restore-remote-triple-quietdatabase-assert

Conversation

@tclinkenbeard-oai

Copy link
Copy Markdown
Collaborator

Summary

A partitioned backup can finish after recovery has selected backup workers for an older transaction-log generation but before those workers read the backup-start key. When the last active backup clears that key, the old-generation workers wait for a future backup instead of acknowledging completion. Their obsolete transaction-log generation remains retained, preventing full cluster recovery.

Allow only workers with a finite old-generation range to exit when no backup is active. This uses the existing acknowledged worker-completion path so the obsolete generation can be retired. Current-generation workers continue waiting for future backups, and active and paused backups retain their existing behavior.

Validation

  • fdbserver_backupworker_test.
  • tests/slow/BackupAndRestore.toml with fault injection and buggification.
  • tests/slow/BackupCorrectnessPartitioned.toml with fault injection and buggification.

@tclinkenbeard-oai
tclinkenbeard-oai marked this pull request as ready for review August 12, 2026 20:01
@foundationdb-ci

Copy link
Copy Markdown
Contributor

Result of foundationdb-pr-clang-ide on Linux RHEL 9

  • Commit ID: 43bc191
  • Duration 0:30:39
  • Result: ✅ SUCCEEDED
  • Error: N/A
  • Build Log terminal output (available for 30 days)
  • Build Workspace zip file of the working directory (available for 30 days)

@foundationdb-ci

Copy link
Copy Markdown
Contributor

Result of foundationdb-pr-clang-arm on Linux RHEL 9

  • Commit ID: 43bc191
  • Duration 0:47:55
  • Result: ✅ SUCCEEDED
  • Error: N/A
  • Build Log terminal output (available for 30 days)
  • Build Workspace zip file of the working directory (available for 30 days)

@foundationdb-ci

Copy link
Copy Markdown
Contributor

Result of foundationdb-pr on Linux RHEL 9

  • Commit ID: 43bc191
  • Duration 1:08:14
  • Result: ✅ SUCCEEDED
  • Error: N/A
  • Build Log terminal output (available for 30 days)
  • Build Workspace zip file of the working directory (available for 30 days)

@foundationdb-ci

Copy link
Copy Markdown
Contributor

Result of foundationdb-pr-clang on Linux RHEL 9

  • Commit ID: 43bc191
  • Duration 1:10:20
  • Result: ✅ SUCCEEDED
  • Error: N/A
  • Build Log terminal output (available for 30 days)
  • Build Workspace zip file of the working directory (available for 30 days)

@foundationdb-ci

Copy link
Copy Markdown
Contributor

Result of foundationdb-pr-cluster-tests on Linux RHEL 9

  • Commit ID: 43bc191
  • Duration 1:30:37
  • Result: ✅ SUCCEEDED
  • Error: N/A
  • Build Log terminal output (available for 30 days)
  • Build Workspace zip file of the working directory (available for 30 days)
  • Cluster Test Logs zip file of the test logs (available for 30 days)

@foundationdb-ci

Copy link
Copy Markdown
Contributor

Result of foundationdb-pr-macos-m1 on macOS 14.x

  • Commit ID: 43bc191
  • Duration 2:50:35
  • Result: ✅ SUCCEEDED
  • Error: N/A
  • Build Log terminal output (available for 30 days)
  • Build Workspace zip file of the working directory (available for 30 days)

@foundationdb-ci

Copy link
Copy Markdown
Contributor

Result of foundationdb-pr-macos on macOS 14.x

  • Commit ID: 43bc191
  • Duration 4:37:20
  • Result: ✅ SUCCEEDED
  • Error: N/A
  • Build Log terminal output (available for 30 days)
  • Build Workspace zip file of the working directory (available for 30 days)

@tclinkenbeard-oai
tclinkenbeard-oai merged commit 58986a8 into apple:main Aug 13, 2026
9 of 10 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants