feat(stargate): reload TLS material without restarts

[mikeyrcamp]

TL;DR

Hot-reload mounted TLS server identities and client trust bundles across Stargate, Pylon, and stargate-k8s-router. Projected certificate and key changes become active without restarting pods, while invalid updates retain the last-known-good configuration.

Additional Details

The affected services previously loaded certificates, private keys, and trust bundles only during startup. Kubernetes could rotate the mounted Secret symlinks, but existing processes continued using stale TLS material until they restarted.

This change adds a shared TLS reload implementation that:

• watches each TLS mount directory for Kubernetes atomic symlink generation changes
• debounces filesystem notifications and retains a five-minute reconciliation poll for missed or unavailable notifications
• validates certificate/key matching, full certificate-chain validity, empty or malformed trust bundles, and a 1 MiB input limit
• publishes a candidate only after the full replacement runtime configuration is ready
• retains the last-known-good material after invalid, incomplete, or inconsistent updates
• replaces client endpoints and closes established connections when trust changes

The reload path is integrated into Stargate direct and reverse tunnels, Pylon direct and reverse tunnels, and stargate-k8s-router Raw QUIC and WebTransport paths. Reload counters, active certificate expiry gauges, expiry-aware readiness, chart guidance, metrics documentation, and an operator rotation runbook are included.

Customer Release Notes

NVCF request-routing components now detect and apply mounted TLS certificate, private-key, and trust-bundle rotations without requiring pod restarts.

Plan Summary

No Kubernetes resources are added or removed. Existing mounted TLS Secrets are watched in place. The chart deployment contract is unchanged.

Usage

Rotate the existing Kubernetes Secret using an atomic Secret update. Keep the certificate and private key in the same Secret. See the Transport TLS Rotation runbook for planned rotation, emergency trust removal, verification, and recovery steps.

For the Reviewer

Please focus on:

• the candidate/commit transaction and generation checks in stargate-tls
• connection replacement and closure on client trust changes
• Pylon readiness remaining closed until initial TLS validation and startup complete
• last-known-good behavior after rejected updates

An independent review found no remaining Critical or Important correctness or security issues.

For QA

Self-managed QA was completed using an image built from this branch. Directory-watcher activation, server identity and client trust rotations, invalid-material rejection, last-known-good retention, readiness, metrics, and pod restart behavior were verified. Established connection replacement and closure are covered by the integration tests.

Testing

Passed:

• the affected Cargo library and binary suites for stargate-tls, stargate, pylon-lib, pylon, and stargate-k8s-router
• ten repeated watcher-suite runs
• a Linux release image build using the inotify backend
• cargo clippy for all affected packages and targets with warnings denied
• cargo fmt --all -- --check
• git diff --check
• llm-request-router Helm lint
• Fern documentation validation
• self-managed control-plane installation with the LLM router and gateway releases deployed
• NVCA installation, rollout, registration, and backend health validation
• live TLS Secret rotation against llm-request-router using the branch-built image

Live rotation results, including Kubernetes Secret projection time:

• a valid generation reloaded in 20 seconds
• a mismatched certificate and key were rejected in 47 seconds while the prior server identity remained active
• a corrected generation reloaded in 13 seconds
• final server identity counters were 2 successful and 1 rejected reload
• readiness remained true and the router pod retained the same identity with zero restarts throughout the in-place updates

Notes

The canonical single-cluster Helmfile BDD flow passed rendering, control-plane installation, release assertions, NVCA registration, rollout, and backend health. The sample function deployment step remains blocked by a test-fixture instance type that the test API does not accept. Test-environment workarounds were limited to unavailable fixtures and images and are not part of this Pull Request.

Issues

Closes #599

References

• feat(stargate): hot reload transport TLS material #599

Related Pull Requests

None.

Dependencies

Adds notify 8.2.0 under CC0-1.0. The license is allowed by the repository policy. No NOTICE update is required.

Checklist

• I am familiar with the Contributing Guidelines.
• I have signed off my commits for Developer Certificate of Origin (DCO) compliance.
• New or existing tests cover these changes.
• The documentation is up to date with these changes.

[coderabbitai]

Important

Review skipped

Draft detected.

Please check the settings in the CodeRabbit UI or the .coderabbit.yaml file in this repository. To trigger a single review, invoke the @coderabbitai review command.

⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Enterprise

Run ID: 9e3c5626-f293-4271-9859-31fa84f4bf56

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

Use the checkbox below for a quick retry:

• 🔍 Trigger review

---

_{Comment @coderabbitai help to get the list of available commands.}

[github-actions]

🌿 Preview your docs: https://nvidia-preview-codex-feat-stargate-tls-hot-reload.docs.buildwithfern.com/nvcf

[github-actions]

🛡️ CodeQL Analysis

🚨 Found 2 issue(s)

Severity Breakdown:

• 🔴 Errors: 0
• 🟡 Warnings: 0
• 🔵 Notes: 0

📋 Top Issues

🔗 View full details in Security tab

_{🕐 Last updated: 2026-08-11 19:41:09 UTC | Commit: fa2e71f}