Skip to content

fix: bound BMP collector shutdown - #1108

Merged
lance0 merged 4 commits into
mainfrom
fix/bmp-reconnect-shutdown
Jul 24, 2026
Merged

fix: bound BMP collector shutdown#1108
lance0 merged 4 commits into
mainfrom
fix/bmp-reconnect-shutdown

Conversation

@lance0

@lance0 lance0 commented Jul 22, 2026

Copy link
Copy Markdown
Owner

Summary

  • stop unreachable BMP collectors promptly while they are connecting or sleeping in reconnect backoff
  • preserve the connected-client shutdown contract: drain manager-queued messages, then send and flush BMP Termination
  • replace per-client relative two-second waits with one absolute two-second client-drain deadline
  • document the bounded shutdown behavior for operators

Root cause

BmpClient observed its message channel only after a successful connection and Initiation. An unreachable collector therefore stayed inside TcpStream::connect and exponential backoff even after the BMP manager had shut down and dropped its senders. The daemon then awaited each client with a fresh two-second timeout, allowing N unreachable collectors to add roughly N × 2 seconds before forced abort.

The fix uses a shared Tokio watch signal only in the pre-Initiation connect/backoff path. Connected clients deliberately ignore it and keep the existing queue-drain, Peer Down ordering, Termination, and flush behavior. After the manager exits, every client join uses the same absolute deadline.

Validation

  • cargo fmt --all -- --check
  • cargo clippy --workspace --all-targets -- -D warnings
  • cargo test --workspace
  • RUSTDOCFLAGS='-D warnings' cargo doc --workspace --no-deps
  • cargo test -p rustbgpd-bmp — 63/63 passed
  • two independent exact-head reviews: SHIP
  • repeated paused-time and loopback focused tests found no flake

Load-bearing regression proof

  • replacing only the cancellation-aware connect select with plain connect.await made the pending-connect assertion fail
  • replacing only the cancellation-aware backoff select with plain sleep made the reconnect-sleep assertion fail
  • deleting only the already-signaled precheck made the subscribed-after-shutdown assertion fail
  • adding eager post-Initiation watch cancellation dropped the queued payload and Termination, making the exact tail assertion fail
  • deleting only the close-path Termination write/flush retained the payload but made the Termination suffix assertion fail
  • restoring the old per-handle relative timeout loop left the three-client helper pending after one two-second budget and made the aggregate-deadline test fail

Linear: LAN-571

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

This PR bounds shutdown latency for BMP collectors by making unreachable/disconnected BMP clients promptly cancelable during connect/backoff, while preserving the existing “connected client drains queued manager messages, then sends/flushes Termination” contract.

Changes:

  • Add a shutdown watch signal used only by disconnected BMP clients to interrupt connect() and reconnect backoff sleeps.
  • Replace per-client 2s join timeouts with a single aggregate client shutdown deadline (2s total).
  • Document the bounded shutdown behavior for operators and record it in the changelog.

Reviewed changes

Copilot reviewed 4 out of 4 changed files in this pull request and generated 1 comment.

File Description
src/main.rs Introduces aggregate client shutdown deadline helper, wires watch signal into clients, and updates shutdown sequencing.
crates/bmp/src/client.rs Implements cancellation-aware connect/backoff loop gated by a watch shutdown signal; adds regression tests.
docs/OPERATIONS.md Documents the bounded BMP shutdown behavior and its operator-visible impact.
CHANGELOG.md Notes the bounded BMP collector shutdown fix under “Fixed”.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Comment thread src/main.rs
Comment on lines +730 to +733
match tokio::time::timeout_at(deadline, &mut client_handles[index]).await {
Ok(Ok(())) => {}
Ok(Err(e)) => warn!(error = %e, "BMP client task panicked during shutdown"),
Err(_) => {
@lance0
lance0 merged commit f9b928a into main Jul 24, 2026
72 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants