bug(routing): withdraw local candidates when their provider gateway is unavailable

[nerdalert]

Summary

In a combined Grid site, provider eligibility currently follows inference-backend health but does not account for availability of the local provider gateway that must serve the candidate. If that gateway is unavailable while its backend remains healthy, the local candidate can remain eligible and local-preference routing can continue selecting an unreachable path instead of failing over to a remote site.

This is production behavior exposed by the combined-site full E2E work, not merely an xtask assertion defect.

Related work:

• feat(security): verify GridSite gateway identity during health evaluation #11 defines identity-aware health evaluation for discovered GridSite gateways. This bug needs the same reachability and identity evidence to affect candidate eligibility, including for the locally declared site.
• Epic: validate Grid demo topologies #17 tracks the combined-site demonstration and its runtime failover proof.
• feat(routing): harden metric-driven provider admission and selection #10 covers backend health and provider admission. Gateway-path health must compose with backend health rather than replace it.

Supported Single-Cluster Topology

A single Kubernetes cluster may run multiple replicas and multiple logical consumer/provider gateways. This provides replica availability, capacity, and policy or credential isolation inside one cluster, while all gateways still share the cluster failure domain.

+------------------------------------------------------------------+
| Kubernetes cluster / one Grid site                               |
|                                                                  |
|  +----------------------+      +----------------------+           |
|  | consumer gateway A   |      | consumer gateway B   |           |
|  | replicas: 2+         |      | tenant/policy split  |           |
|  +----------+-----------+      +----------+-----------+           |
|             |                             |                       |
|             +--------- Grid overlay ------+                       |
|                           |                                      |
|              +------------+------------+                         |
|              |                         |                         |
|              v                         v                         |
|  +----------------------+  +----------------------+               |
|  | provider gateway A   |  | provider gateway B   |               |
|  | replicas: 2+         |  | separate identity,   |               |
|  | credential set A     |  | policy + credentials |               |
|  +----------+-----------+  +----------+-----------+               |
|             |                         |                           |
|             v                         v                           |
|  +----------------------+  +----------------------+               |
|  | vLLM/backend pool A  |  | backend/API pool B   |               |
|  +----------------------+  +----------------------+               |
|                                                                  |
|  Grid operator publishes only candidates whose complete serving  |
|  path is eligible: backend + provider gateway + policy/trust.    |
+------------------------------------------------------------------+

Multiple replicas of one gateway provide pod/node-level availability. Multiple logical gateways can isolate provider groups, tenants, credentials, identities, and policy. Neither arrangement provides cluster-level fault isolation; that requires multiple clusters/sites.

Reproduction

In the three-cluster combined-site topology:

1. Confirm west, central, and east each advertise a healthy mock-model candidate.
2. Confirm the central consumer selects the central candidate through the central provider gateway.
3. Scale provider-gateway in central to zero while leaving mock-inference-central healthy.
4. Observe that the central candidate can remain present and eligible because backend health remains successful.
5. Send a new request through the central consumer.
6. Local preference may continue selecting central rather than an eligible west/east provider path.

Draining mock-inference-central does withdraw the candidate and remote fallback works after overlay convergence. That is a valid provider-capacity test but does not resolve this gateway-outage bug.

Expected Behavior

A candidate is eligible only when the complete serving path is eligible:

backend healthy
AND provider gateway reachable
AND expected gateway identity authenticated
AND gateway/site policy permits the path
AND health evidence is fresh

If every gateway instance serving a candidate becomes unavailable or fails identity verification, Grid must withdraw or mark that candidate ineligible. Consumers should receive the updated overlay and route new eligible traffic to another site according to policy.

For multiple gateways in one cluster, health must be evaluated against the gateway associated with each candidate or provider group. Failure of gateway A must not withdraw candidates served exclusively by healthy gateway B.

Acceptance Criteria

• Model the association between a candidate/provider and its serving provider gateway without relying on ambiguous site-wide defaults.
• Incorporate bounded, fresh provider-gateway availability into candidate eligibility.
• Reuse the authenticated gateway-health contract from feat(security): verify GridSite gateway identity during health evaluation #11 rather than treating TCP reachability alone as sufficient.
• A local provider gateway outage withdraws only candidates whose serving path depends on that gateway.
• A healthy backend behind an unavailable gateway is not advertised as routable.
• Multiple replicas behind one provider-gateway Service remain eligible while at least one verified serving replica/path is available.
• Multiple provider gateways in one cluster are evaluated independently.
• New sessions fail over to an identified eligible remote provider after overlay convergence.
• Recovery republishes the candidate and restores local preference without restarting consumer gateways.
• Existing-session behavior during gateway loss is explicitly defined and tested.
• Stale or missing gateway-health evidence fails closed after a bounded grace period.
• Status, Events, logs, metrics, and evidence use bounded reason codes and expose no credentials, certificate private material, or internal topology through public response headers.
• Runtime tests cover gateway Deployment scale-to-zero, Service endpoint loss, wrong gateway identity, recovery, and the multi-gateway isolation case shown above.

Non-Goals

• Claiming that multiple gateways in one Kubernetes cluster provide cluster-level or site-level disaster recovery.
• Replacing backend health, capacity, or admission signals.
• Making the Grid operator proxy inference traffic.
• Exposing raw endpoint inventories or private transport topology to callers.

[nerdalert]

Broader admission and recovery semantics are tracked in #10. This issue should remain focused on associating each candidate with its serving provider gateway and withdrawing that candidate when the complete gateway path is unavailable.\n\nImplementation should use the admission ordering, stale/missing evidence, hysteresis, recovery, and bounded-reason contract defined by #10 rather than creating a gateway-specific state model with different semantics.