feat(routing): publish provider selection modes

[nerdalert]

Summary

This PR adds an explicit provider-selection contract to Grid. It accompanies the research spike in grid#31 (#31) to gather feedback and converge on an approach to provider-level load distribution.

Its scope is selecting among provider gateways, endpoint selection within a provider remains the responsibility of that provider’s serving stack.

Grid determines which provider gateways are eligible, organizes equivalent providers into selection groups, and publishes the selected distribution mode in the routing overlay. Praxis consumes that immutable overlay and performs request-time selection locally.

  Client
    |
    v
  Consumer Gateway
    |
    v
  intelligent_route
    |
    +--> Provider Gateway A
    +--> Provider Gateway B
    `--> Provider Gateway C

This operates at the provider-gateway layer. Endpoint selection within a provider, including llm-d EPP backend selection, remains the responsibility of that provider's serving stack.

Architecture

Provider routing is separated into four stages:

  Health, authorization, admission, and freshness
                          |
                          v
                  Routing policy
                          |
                          v
                   Scoring policy
                          |
                          v
                 Selection groups
                          |
                          v
                  Selection mode

Routing Policy

geographyFirst groups providers by admission, locality, and freshness. Praxis can balance across providers in the closest viable locality tier while keeping remote tiers available as fallback.

scoreFirst groups providers by admission and freshness. Eligible providers from multiple sites can therefore participate in the same active group.

Scoring Policy

The scoring strategies remain independent of request distribution:

• noMetrics: No inference metrics are required; routing continues to use health, admission, freshness, locality, and authorization.
• queueDepth: Grid retrieves queue-depth metrics from the provider’s llm-d EPP metrics endpoint and uses them to rank provider gateways by queue pressure.
• kvCachePressure: Grid retrieves KV-cache utilization metrics from the provider’s llm-d EPP metrics endpoint and uses them to rank provider gateways by available cache capacity.

A score difference does not become a traffic ratio and does not split otherwise equivalent providers into different selection groups.

Selection Policy

spec:
routingPolicy: scoreFirst
scoringPolicy:
strategy: noMetrics
selectionPolicy:
mode: roundRobin

deterministic selects the first viable provider in the active group. This is useful when a deployment requires a strict preferred provider.

roundRobin rotates equally among viable providers in the active group. It provides active/active provider distribution without requiring EPP, Prometheus, or inference-specific metrics.

Selection policy is explicit. Grid does not infer it from scores, provider count, routing policy, or the availability of metrics.

Responsibilities

Grid remains the asynchronous control plane:

• observes provider health, admission, freshness, and configured metrics;
• applies routing and scoring policies;
• creates deterministic selection groups;
• publishes immutable routing overlays;
• removes or restricts unavailable providers through admission state.

Praxis remains the request-time data plane:

• loads and validates an overlay snapshot;
• evaluates session affinity before new-provider selection;
• selects locally within the first viable group;
• preserves existing bindings when admission permits existing traffic;
• falls through to the next viable group when the active group has no eligible provider.

The request path does not call Grid, Kubernetes, EPP, Prometheus, ConfigMaps, or another remote control-plane service.

Multiple Consumer Gateways

The contract supports multiple consumer gateways. Each consumer receives the same Grid-produced provider ordering and group metadata, then performs selection from its local in-memory snapshot.

Round-robin counters are local to each gateway process. A single gateway produces a predictable local sequence, but multiple replicas do not share a global counter. Aggregate traffic should converge toward an even distribution over a sufficient request volume, but strict global request-by-request ordering is not guaranteed.

Session affinity can also make the observed distribution intentionally uneven because an existing session remains bound to its selected provider.

Compatibility

selectionPolicy is optional.

When it is omitted, Grid omits selection_policy from the generated overlay. Consumers that do not receive an explicit mode retain deterministic selection
behavior.

New Helm installations can explicitly select roundRobin, while existing resources with an omitted policy are not silently changed.

The field is additive across the Grid, overlay-sync, and Praxis contracts.

Overlay Contract

Grid publishes:

  {
    "selection_policy": {
      "mode": "roundRobin"
    },
    "candidates": [
      {
        "cluster": "provider-a",
        "selection_group": 0
      },
      {
        "cluster": "provider-b",
        "selection_group": 0
      },
      {
        "cluster": "provider-c",
        "selection_group": 0
      }
    ]
  }

Selection groups are:

• deterministic under candidate input permutations;
• scoped independently by capability;
• contiguous and ordered by preference;
• separated by admission and routing-policy boundaries;
• unaffected by candidate identity alone;
• unaffected by metric score differences within an otherwise equivalent group.

The selection policy and group metadata are included in semantic overlay validation and digest calculation.

Components

• Grid feature branch (https://github.com/nerdalert/grid/tree/feat/provider-selection-groups)

• Provider selection architecture
  (https://github.com/nerdalert/grid/blob/feat/provider-selection-groups/docs/architecture/provider-selection-and-load-balancing.md)

• Companion Praxis AI selection branch (https://github.com/nerdalert/ai/tree/feat/provider-load-balancing)

• Grid provider-selection research spike and demo video (Spike: request-time selection across equivalent Grid providers #31)

• Public Grid demos (https://github.com/praxis-proxy/demos)

• Praxis tracing visualization (https://github.com/nerdalert/praxis-tracing)

The companion AI change implements the in-memory group index and request-time selection used by intelligent_route. The Grid and AI changes preserve a clean
control-plane/data-plane boundary.

OpenTelemetry instrumentation and the tracing UI are separate components. They improve visualization and validation but are not required for provider selection to operate.

Scope

This PR includes:

• optional selectionPolicy.mode;
• deterministic and roundRobin modes;
• deterministic Grid-side selection groups;
• overlay contract and digest updates;
• overlay-sync validation;
• CRD, Helm, sample, and architecture documentation updates;
• provider-traffic demo orchestration.

This PR does not include:

• weighted request selection;
• metric-derived traffic weights;
• synchronized counters across gateway replicas;
• request-time metric queries;
• backend endpoint selection inside a provider;
• OpenTelemetry instrumentation.

Weighted selection remains a separate extension. It can build on the same active-group boundary without changing the distinction between scoring and request distribution.

Validation

Static validation passed:

• make lint
• make test
• Rust documentation with warnings denied
• focused operator tests
• focused overlay-sync tests
• focused xtask tests
• image-override tests
• CRD parity checks
• Helm rendering and schema rejection checks
• git diff --check

The focused cold Kind proof validated:

• three provider clusters;
• one consumer gateway;
• three provider gateways discovered through SWIM;
• scoreFirst + noMetrics + roundRobin;
• 60 of 60 successful requests;
• exact 20/20/20 provider distribution;
• a repeating three-provider sequence;
• stable overlay and serving revisions during traffic;
• successful teardown.

The demo narrative and recording are available in grid#31 (#31).

The focused provider-traffic demonstration uses one consumer gateway and three provider gateways:

  Client
    |
    v
  Consumer gateway
    |
    +--> Provider gateway A --> VCR backend A
    +--> Provider gateway B --> VCR backend B
    `--> Provider gateway C --> VCR backend C

Run it with the published images (a demo PR is not opened yet for reference but the validation images are published):

  - ghcr.io/nerdalert/praxis-ai:grid-provider-selection-otel-20260814-536534aba5c1
  - ghcr.io/nerdalert/grid-operator:provider-selection-20260814-d965540dad8d
  - ghcr.io/nerdalert/grid-overlay-sync:provider-selection-20260814-d965540dad8d
  - ghcr.io/nerdalert/praxis-tracing:provider-selection-20260814-fbadb3e2007a

  source demos/grid-provider-traffic/configs/images.env
  export GRID_XTASK_VCR_IMAGE=ghcr.io/neuralmagic/vllm-vcr:vllm0.23
  export GRID_XTASK_IMAGE_PULL_POLICY=IfNotPresent
  export GRID_REPO=/path/to/grid

  ./demos/grid-provider-traffic/run.sh --quick --teardown

The expected proof is 60 successful requests with exact 20/20/20 provider attribution and a repeating three-provider sequence. The demo uses noMetrics and roundRobin; it does not claim weighted routing or metric-derived traffic ratios.

Operational Considerations

Provider membership and eligibility are eventually consistent because Grid updates overlays asynchronously.

A gateway begins using new membership only after overlay-sync validates the overlay and Praxis accepts the corresponding serving revision. Selection state is scoped to the loaded snapshot and may restart when routing semantics genuinely change.

The operator avoids rewriting semantically unchanged overlay ConfigMaps. This prevents metadata-only reconciliation from resetting request-time selection state while retaining repair behavior for malformed, incomplete, or corrupted payloads.

Breaking Changes

No breaking change is intended.

Deployments that omit selectionPolicy retain deterministic behavior. Round-robin distribution is enabled only when explicitly configured.

Related Components

This change is part of a cross-repository spike. The following feature branches are not yet merged:

• Grid provider-selection branch (https://github.com/nerdalert/grid/tree/feat/provider-selection-groups)
  Defines selection policy, constructs provider groups, publishes the overlay contract, and supplies the demo orchestration.

• Praxis AI provider-selection branch (https://github.com/nerdalert/ai/tree/feat/provider-load-balancing)
  Consumes the overlay, builds the in-memory group index, and performs request-time selection in intelligent_route.

The following resources provide supporting context and validation:

• Grid issue Spike: request-time selection across equivalent Grid providers #31 (Spike: request-time selection across equivalent Grid providers #31)
  Tracks the research spike and contains the focused demo narrative and video.

• Public Grid demos (https://github.com/praxis-proxy/demos)
  Contains the reusable demo framework. The provider-traffic additions associated with this spike may not yet be available on its default branch.

• Praxis tracing visualization (https://github.com/nerdalert/praxis-tracing) Provides optional visualization and trace inspection. It is not required for provider selection.

The Grid and Praxis AI changes must be evaluated together for the complete feature. This Grid PR defines and publishes the contract; the companion AI branch implements its request-time consumer. OpenTelemetry and the tracing UI remain optional supporting components.

Performance note

The grouping pass is O(n) in the asynchronous control-plane render path. It does not run on the request path, so a separate benchmark is not required for this change; workspace lint, tests, and the focused provider-selection validation cover the affected behavior.

[jordigilh]

Comprehensive GA-readiness pass on the routing/grouping contract, test coverage, and Helm templating. Nothing here blocks the design; flagging a few things worth closing before this leaves draft.

[nerdalert]

Addressed the review feedback:

• Documented that the updated geographyFirst ordering can change the first deterministic provider during an upgrade, even when selectionPolicy is omitted.
• Corrected the stale freshness/locality comment.
• Added Helm coverage for the new GridNetwork roundRobin default.
• Added the requested performance note: grouping is O(n) in the asynchronous control-plane render path and is not request-path work, so a separate benchmark is not required for this change.

The branch was rebased onto current main and the changes passed make lint, operator tests, formatting, diff checks, CRD parity, Helm lint with valid values, and Helm rendering.

[praxis-bot]

PR Review

Purpose: Adds an explicit selectionPolicy field to GridNetwork with deterministic, roundRobin, and random modes. Introduces deterministic selection-group assignment, propagates the policy through the overlay and consumer config, and includes a 3000-line provider-traffic demo orchestrator.

Assessment: Well-structured, backward-compatible feature addition. CRD, overlay, overlay-sync, and Helm changes are consistent. Two test infrastructure gaps should be addressed before merge.

Severity	Count
Medium	2

Non-inline findings

[Medium] tests/fixtures/overlay-contract/v1/manifest.json does not list the new valid-selection-policy.json fixture. The fixture_manifest_digests_are_correct test in overlay_envelope.rs iterates manifest entries and verifies each fixture's digest. Because valid-selection-policy.json is missing from the manifest, this loop does not cover it. The fixture is tested directly in validation.rs::selection_policy_fixture_digest_matches_operator_contract, but adding it to the manifest keeps the manifest authoritative and ensures the envelope-side fixture sweep covers it too.

[praxis-bot]

[Medium] The overlay_digest helper (line 492) constructs its semantic payload without selection_policy, diverging from the real implementations in overlay_envelope.rs::compute_semantic_digest and validation.rs::compute_raw_semantic_digest. Currently harmless because test_overlay() sets selection_policy: None, but any future test that sets a non-None selection_policy will compute an incorrect digest and fail with an obscure mismatch. Consider adding the selection_policy branch to overlay_digest to keep it in sync with the production digest.