fix: bound BMP collector shutdown - #1108
Merged
Merged
Conversation
There was a problem hiding this comment.
Pull request overview
This PR bounds shutdown latency for BMP collectors by making unreachable/disconnected BMP clients promptly cancelable during connect/backoff, while preserving the existing “connected client drains queued manager messages, then sends/flushes Termination” contract.
Changes:
- Add a shutdown watch signal used only by disconnected BMP clients to interrupt
connect()and reconnect backoff sleeps. - Replace per-client 2s join timeouts with a single aggregate client shutdown deadline (2s total).
- Document the bounded shutdown behavior for operators and record it in the changelog.
Reviewed changes
Copilot reviewed 4 out of 4 changed files in this pull request and generated 1 comment.
| File | Description |
|---|---|
| src/main.rs | Introduces aggregate client shutdown deadline helper, wires watch signal into clients, and updates shutdown sequencing. |
| crates/bmp/src/client.rs | Implements cancellation-aware connect/backoff loop gated by a watch shutdown signal; adds regression tests. |
| docs/OPERATIONS.md | Documents the bounded BMP shutdown behavior and its operator-visible impact. |
| CHANGELOG.md | Notes the bounded BMP collector shutdown fix under “Fixed”. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
Comment on lines
+730
to
+733
| match tokio::time::timeout_at(deadline, &mut client_handles[index]).await { | ||
| Ok(Ok(())) => {} | ||
| Ok(Err(e)) => warn!(error = %e, "BMP client task panicked during shutdown"), | ||
| Err(_) => { |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Root cause
BmpClientobserved its message channel only after a successful connection and Initiation. An unreachable collector therefore stayed insideTcpStream::connectand exponential backoff even after the BMP manager had shut down and dropped its senders. The daemon then awaited each client with a fresh two-second timeout, allowing N unreachable collectors to add roughly N × 2 seconds before forced abort.The fix uses a shared Tokio watch signal only in the pre-Initiation connect/backoff path. Connected clients deliberately ignore it and keep the existing queue-drain, Peer Down ordering, Termination, and flush behavior. After the manager exits, every client join uses the same absolute deadline.
Validation
cargo fmt --all -- --checkcargo clippy --workspace --all-targets -- -D warningscargo test --workspaceRUSTDOCFLAGS='-D warnings' cargo doc --workspace --no-depscargo test -p rustbgpd-bmp— 63/63 passedLoad-bearing regression proof
connect.awaitmade the pending-connect assertion failLinear: LAN-571