Restart backup workers after disk I/O timeouts

[tclinkenbeard-oai]

Summary

• Recover backup-worker I/O timeouts without bypassing storage-server restart safety.
• Re-recruit failed old-generation classic and range-partitioned backup workers after acknowledged owner shutdown; ambiguous failures safely fall back to transaction-system recovery.
• Publish backup start versions transactionally with old-log retirement, and add focused classic and range-partitioned lifecycle, ownership, and retention regressions.

Rollout

• Full protection against the pending-backup retirement race requires both backup agents and server processes to run the updated code.

Validation

• fdbclient_test
• fdbserver_backupworker_test
• fdbserver_clustercontroller_test
• fdbserver_logsystem_test
• fdbserver_worker_test
• tests/slow/BackupAndRestore.toml
• tests/slow/BackupCorrectnessPartitioned.toml

All simulation configurations were run with fault injection and buggification enabled.

[foundationdb-ci]

Result of foundationdb-pr-clang-ide on Linux RHEL 9

• Commit ID: 41c703c
• Duration 0:41:42
• Result: ✅ SUCCEEDED
• Error: N/A
• Build Log terminal output (available for 30 days)
• Build Workspace zip file of the working directory (available for 30 days)

[foundationdb-ci]

Result of foundationdb-pr-clang-arm on Linux RHEL 9

• Commit ID: 41c703c
• Duration 0:46:08
• Result: ✅ SUCCEEDED
• Error: N/A
• Build Log terminal output (available for 30 days)
• Build Workspace zip file of the working directory (available for 30 days)

[foundationdb-ci]

Result of foundationdb-pr-clang on Linux RHEL 9

• Commit ID: 41c703c
• Duration 1:14:21
• Result: ✅ SUCCEEDED
• Error: N/A
• Build Log terminal output (available for 30 days)
• Build Workspace zip file of the working directory (available for 30 days)

[foundationdb-ci]

Result of foundationdb-pr-cluster-tests on Linux RHEL 9

• Commit ID: 41c703c
• Duration 1:39:02
• Result: ✅ SUCCEEDED
• Error: N/A
• Build Log terminal output (available for 30 days)
• Build Workspace zip file of the working directory (available for 30 days)
• Cluster Test Logs zip file of the test logs (available for 30 days)

[foundationdb-ci]

Result of foundationdb-pr-macos-m1 on macOS 14.x

• Commit ID: 41c703c
• Duration 4:12:16
• Result: ❌ FAILED
• Error: `Error while executing command: # compatible with bash and zsh
  ssh -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null -i ${HOME}/.ssh_key
  -o ServerAliveInterval=9 ec2-user@${MAC_EC2_HOST} "
  set -e -o noclobber
  BUILDNUM=${CODEBUILD_BUILD_NUMBER}
  WAIT_TIME=14400 # 4 hours
  LOCK_STALE=4500 # 75 minutes
  LOCK=/tmp/ci.lock
  "'

macOS stat flags, not linux compatible

lock_mtime() { stat -f %m $LOCK || echo 0 }
I=0
while ! (echo $BUILDNUM >$LOCK) 2>/dev/null ; do
if (( ++I >= WAIT_TIME )); then
echo "timeout waiting for ci.lock after $I seconds"
exit 1
fi
if (( $(date +%s) - $(lock_mtime) >= LOCK_STALE )); then
echo "lock expired after $LOCK_STALE seconds ..."
# racy but sleep so chances are low
sleep $(( 3 + RANDOM % 8 ))
if (( $(date +%s) - $(lock_mtime) >= LOCK_STALE )); then
echo "lock holder is gone"
rm -f $LOCK
else
echo "new lock holder"
fi
fi
sleep 1
done
echo "acquired ci.lock"
'
. Reason: exit status 1`

• Build Log terminal output (available for 30 days)
• Build Workspace zip file of the working directory (available for 30 days)

[foundationdb-ci]

Result of foundationdb-pr-macos on macOS 14.x

• Commit ID: 41c703c
• Duration 4:12:37
• Result: ❌ FAILED
• Error: `Error while executing command: # compatible with bash and zsh
  ssh -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null -i ${HOME}/.ssh_key
  -o ServerAliveInterval=9 ec2-user@${MAC_EC2_HOST} "
  set -e -o noclobber
  BUILDNUM=${CODEBUILD_BUILD_NUMBER}
  WAIT_TIME=14400 # 4 hours
  LOCK_STALE=4500 # 75 minutes
  LOCK=/tmp/ci.lock
  "'

macOS stat flags, not linux compatible

lock_mtime() { stat -f %m $LOCK || echo 0 }
I=0
while ! (echo $BUILDNUM >$LOCK) 2>/dev/null ; do
if (( ++I >= WAIT_TIME )); then
echo "timeout waiting for ci.lock after $I seconds"
exit 1
fi
if (( $(date +%s) - $(lock_mtime) >= LOCK_STALE )); then
echo "lock expired after $LOCK_STALE seconds ..."
# racy but sleep so chances are low
sleep $(( 3 + RANDOM % 8 ))
if (( $(date +%s) - $(lock_mtime) >= LOCK_STALE )); then
echo "lock holder is gone"
rm -f $LOCK
else
echo "new lock holder"
fi
fi
sleep 1
done
echo "acquired ci.lock"
'
. Reason: exit status 1`

• Build Log terminal output (available for 30 days)
• Build Workspace zip file of the working directory (available for 30 days)

[foundationdb-ci]

Result of foundationdb-pr on Linux RHEL 9

• Commit ID: 41c703c
• Duration 0:48:34
• Result: ❌ FAILED
• Error: Error while executing command: ctest -j ${NPROC} --no-compress-output -T test --output-on-failure. Reason: exit status 8
• Build Log terminal output (available for 30 days)
• Build Workspace zip file of the working directory (available for 30 days)