feat(token-quota): add distributed quota topology

[nerdalert]

Summary

This PR adds the Grid-side provider-selection contract used by Praxis AI for request-time load distribution and introduces a Forge topology for validating distributed token quotas alongside Grid routing.

Grid remains the asynchronous control plane:

  Provider health and telemetry
              |
              v
         Grid Operator
    - determines admission
    - applies routing policy
    - calculates preference
    - creates selection groups
    - publishes selection mode
              |
              v
        Routing overlay
              |
              v
         overlay-sync
              |
              v
          Praxis AI
    - loads immutable snapshot
    - preserves session affinity
    - selects within active group
    - performs no Grid call per request

This PR does not add token quota enforcement to Grid. The included quota topology demonstrates that Praxis AI can enforce one shared quota while Grid independently distributes admitted requests across provider gateways.

Project Stack

  Praxis core
    Establishes trusted authenticated identity
    identity.user_id
          |
          v
  Praxis AI
    Applies token quota admission
    Rejects exhausted requests before routing
          |
          v
  Grid control plane
    Publishes eligible providers, priority groups,
    and the configured selection mode
          |
          v
  Praxis AI intelligent_route
    Selects a provider from its in-memory overlay
          |
          v
  Provider gateway and inference stack

The responsibility boundaries are explicit:

• Praxis core authenticates callers and establishes trusted identity metadata.
• Praxis AI owns quota enforcement and request-time provider selection.
• Grid owns provider admission, routing preference, selection groups, and overlay publication.
• overlay-sync validates and delivers the published routing state.
• The provider stack executes admitted inference requests.
• Valkey, when configured by AI, stores shared quota state across gateway replicas.

Grid does not read quota state, contact Valkey, or place quota enforcement in its reconciliation loop.

Related Work

• AI provider-selection foundation: feat(routing): add provider load balancing, feat(routing): add provider load balancing ai#731 (feat(routing): add provider load balancing ai#731)
• Grid provider-selection research: Spike: request-time selection across equivalent Grid providers #31 (Spike: request-time selection across equivalent Grid providers #31)
• Distributed token quota demo: grid-distributed-token-rate-limit (https://github.com/praxis-proxy/experimental/tree/main/demos/grid-distributed-token-rate-limit)

The AI PR implements request-time selection from the Grid-produced contract. This Grid PR publishes that contract and supplies the integration topology. The distributed quota behavior is demonstrated separately in the experimental repository.

Provider-Selection Contract

The GridNetwork API gains an explicit selection policy:

  spec:
    selectionPolicy:
      mode: roundRobin

Supported modes are:

• deterministic
• roundRobin
• random

The rendered routing overlay carries the resolved mode using the wire contract consumed by Praxis AI:

  {
    "selection_policy": {
      "mode": "roundRobin"
    }
  }

Each candidate can also carry a selection group:

  {
    "cluster": "provider-a",
    "selection_group": 0
  }

The two fields answer different questions:

selection_group
Which candidates may actively share traffic?

selection_policy.mode
How should Praxis choose among candidates in that group?

Grid determines the groups. Praxis performs the selected operation locally on the request path.

Selection Modes

Deterministic

  selectionPolicy:
    mode: deterministic

Praxis chooses the first viable candidate in the best active group.

This supports strict provider preference, ordered failover, and configurations where one provider should receive new traffic until it becomes ineligible.

Round Robin

  selectionPolicy:
    mode: roundRobin

Praxis takes turns across viable candidates in the best active group.

This provides equal active-active distribution without requiring EPP, Prometheus, or inference-specific metrics.

Stacked on Provider Selection

This branch is stacked on Grid PR #65, the provider-selection foundation.

PR #65 provides:

• provider selection groups;
• selectionPolicy.mode;
• deterministic, roundRobin, and random selection modes;
• overlay and Helm support for the selection contract.

This branch adds only the distributed token-quota integration topology and the gateway configuration needed to validate it. It does not duplicate or modify the provider-selection implementation from PR #65.

The quota topology exercises:

• two independently addressable consumer gateways;
• shared Valkey-backed quota state;
• three provider gateways;
• Grid’s roundRobin provider-selection contract;
• Praxis AI quota admission before provider routing.

Quota enforcement remains owned by Praxis AI. Grid publishes provider routing state but does not read Valkey, reserve tokens, or enforce quota.

Random

  selectionPolicy:
    mode: random

Praxis selects uniformly among viable candidates in the best active group.

This provides stateless equal-probability distribution without converting Grid scores into traffic weights.

Routing and Scoring Policies

Selection remains separate from routing and scoring.

Geography First

geographyFirst groups candidates using:

• admission state;
• locality;
• freshness.

Praxis can balance among equivalent providers in the closest viable locality tier. Remote providers remain available in lower-priority groups.

  Group 0: healthy local providers
  Group 1: healthy remote providers
  Group 2: degraded fallback providers

Score First

scoreFirst groups candidates using:

• admission state;
• freshness.

Eligible providers from different sites can therefore share the active group.

Scoring

Existing scoring strategies remain independent from request distribution:

• noMetrics
• queueDepth
• kvCachePressure

For metric-aware strategies, Grid asynchronously obtains the configured signals from the llm-d EPP Prometheus-compatible metrics endpoint.

Scores affect ordering and observability. A score difference:

• does not become a traffic ratio;
• does not implicitly enable weighted routing;
• does not split otherwise equivalent candidates into different selection groups.

For example:

  scoringPolicy:
    strategy: queueDepth

  selectionPolicy:
    mode: roundRobin

Queue pressure may affect provider preference and group ordering, while traffic remains equally distributed within the active group.

Selection-Group Construction

Selection groups are assigned after the existing deterministic candidate sort.

Groups are scoped independently by capability and capability name. Candidates for different models or tools do not share selection state.

The grouping rules preserve operational boundaries:

• excluded providers cannot receive traffic;
• existing_only providers cannot receive new sessions;
• freshness boundaries remain intact;
• geography boundaries are preserved by geographyFirst;
• capability boundaries remain isolated;
• lower-priority groups remain fallback groups.

Candidate identity is not itself a grouping boundary. Multiple equivalent provider gateways can therefore share one active group.

Scores remain available for ordering and explanation, but metric noise does not create a new group on every score change.

Session Affinity and Admission

Grid publishes provider admission state as part of the candidate contract.

Praxis applies that state as follows:

• new_and_existing: eligible for new requests and existing bindings;
• existing_only: excluded from new selection but reusable by an existing session;
• excluded: unavailable for both new and existing traffic.

Session affinity is evaluated before the configured selection mode.

  Request
     |
     v
  Existing valid session binding?
     | yes
     +------> reuse bound provider
     |
     no
     v
  Select from best viable group

A provider can therefore stop receiving new sessions without immediately breaking valid existing sessions.

Overlay Integrity

The selection policy and candidate selection groups are part of the semantic routing contract.

They are included in overlay digest calculation so that changes to request-distribution behavior produce a new semantic revision.

The operator and overlay-sync agree on the digest-significant payload:

• network identity;
• local site;
• candidates;
• candidate selection groups;
• selection policy when present.

Overlay-sync validates the raw semantic JSON rather than reconstructing candidates through an older typed representation. This preserves additive fields for forward-compatible consumers.

Credential and Secret-reference structures remain strict and continue to reject unknown security-sensitive fields.

Compatibility

selectionPolicy is optional in the Grid CRD.

When it is omitted, the operator omits selection_policy from the generated overlay. Consumers that do not understand the new fields can ignore the additive candidate metadata, while
compatible Praxis AI versions retain deterministic behavior for an omitted selection policy.

New deployments can explicitly select round-robin behavior:

gridNetwork:
selectionPolicy:
mode: roundRobin

The policy is not inferred from:

• provider count;
• candidate rank;
• score values;
• scoring strategy;
• the presence of inference metrics;
• selection-group metadata.

This avoids silently changing traffic distribution during a partial upgrade.

The Helm chart continues to render provider admissionPolicy independently of selectionPolicy. Admission stabilization and request distribution are separate controls and can be configured together.

Multiple Consumer Gateways

The overlay contract can be consumed by multiple Praxis consumer gateways.

  Consumer Gateway A ─┐
                      ├── same Grid routing contract
  Consumer Gateway B ─┘

Each gateway performs selection from its own in-memory snapshot. No centralized request-time coordinator is required.

Important behavior:

• Each gateway has independent round-robin state.
• Aggregate distribution converges toward equal traffic as request volume grows.
• The gateways do not maintain one globally synchronized round-robin counter.
• Session affinity can intentionally make distribution uneven.
• Restarts or semantic snapshot replacement reset local selection state.
• Provider eligibility remains consistent when consumers serve the same overlay revision.

This design avoids placing Grid, Kubernetes, or a distributed counter in the request hot path.

Distributed Token-Quota Topology

The PR includes a Forge topology for exercising shared token quota enforcement together with Grid provider selection.

The topology contains:

                    Client
                      |
            +---------+---------+
            |                   |
            v                   v
    Consumer Gateway A  Consumer Gateway B
            |                   |
            +---------+---------+
                      |
            Shared AI quota ledger
                  in Valkey
                      |
            admitted requests only
                      |
                      v
               Grid routing overlay
                      |
         +------------+------------+
         |            |            |
         v            v            v
    Provider West Provider Central Provider East

The two consumer gateways share one Valkey-backed quota. Grid publishes the three eligible providers and round-robin selection mode.

The topology proves that:

• quota remains shared across gateway processes;
• admitted requests can be distributed across provider gateways;
• quota identity does not change when the selected provider changes;
• exhausted requests receive 429 before provider selection;
• denied requests do not reach a provider backend;
• Grid remains independent from quota enforcement.

The topology uses Forge’s existing manifest, Helm, template, and readiness primitives. It does not introduce quota functionality into the Grid operator.

Image Configuration

The topology must not depend on contributor-owned registry images.

Gateway, operator, overlay-sync, tracing, and supporting images are supplied through explicit configuration and the existing image-override mechanism.

Example local validation:

  export GRID_XTASK_GATEWAY_IMAGE=praxis-ai:token-rate-limit-demo
  export GRID_XTASK_OPERATOR_IMAGE=grid-operator:token-rate-limit-demo
  export GRID_XTASK_OVERLAY_SYNC_IMAGE=grid-overlay-sync:token-rate-limit-demo
  export GRID_XTASK_IMAGE_PULL_POLICY=Never

Published demonstrations may provide pinned images separately, but Grid repository defaults remain organization-owned or explicitly overridden.

Security Considerations

Provider selection does not consume credentials, request bodies, prompts, completions, authorization headers, or raw session identifiers.

The quota topology uses Secret-backed configuration for:

• Basic Auth credentials;
• Valkey authentication;
• provider trust material;
• provider credentials.

The topology is intended for integration validation, not as a production security reference. Production deployments should use:

• encrypted Valkey transport;
• independently managed credentials;
• certificate validation;
• namespace-scoped RBAC;
• restrictive NetworkPolicies;
• production identity providers;
• external Secret management where appropriate.

No raw user identity, quota key, credential, request ID, or trace ID is introduced as a Prometheus label by the Grid changes.

Performance and Hot-Path Behavior

All Grid work remains asynchronous.

Grid may:

• observe health and metrics;
• compute admission state;
• order candidates;
• assign selection groups;
• render routing overlays;
• publish semantic revisions.

Praxis performs request-time selection from an already loaded in-memory snapshot.

This PR does not add the following to the request path:

• Grid API calls;
• Kubernetes API calls;
• ConfigMap reads;
• Prometheus scrapes;
• EPP requests;
• operator reconciliation;
• remote scoring calls;
• distributed round-robin counters;
• quota calls owned by Grid.

The group index and selection state belong to Praxis AI and are constructed when the overlay is loaded.

Observability

The overlay exposes bounded routing context suitable for logs, metrics, and traces:

• admission state;
• rank;
• selection group;
• selection mode;
• locality tier;
• provider identity;
• serving overlay revision.

This allows an operator to distinguish:

  Why was this provider eligible?
    -> admission, freshness, locality, and scoring

  Why did this request use this provider?
    -> active group, selection mode, affinity, and local selection state

The token-quota demo can additionally show the AI-owned admission result, remaining quota, selected provider for admitted requests, and the absence of a provider hop for denied requests.

Documentation

The provider-selection architecture is documented in the repository and covers:

• responsibility boundaries;
• routing, scoring, and selection policies;
• selection-group construction;
• session affinity;
• admission states;
• multiple consumer gateways;
• compatibility behavior;
• digest semantics;
• operational limitations;
• future weighted selection.

The research demonstration and video are linked from Grid issue #31 (#31).

Current Scope

This PR implements:

• optional selectionPolicy.mode;
• deterministic, round-robin, and random mode contracts;
• deterministic selection-group assignment;
• semantic-digest coverage;
• overlay-sync validation;
• Helm and CRD configuration;
• provider-traffic Forge validation;
• distributed token-quota Forge topology.

It does not implement:

• request-time selection inside Grid;
• synchronized counters across consumer gateways;
• weighted-random provider selection;
• metric-derived traffic weights;
• quota storage or enforcement in Grid;
• direct inference-engine metric scraping;
• dynamic quota policy management;
• a production tracing backend.

Weighted provider distribution remains a separate extension to the selection contract.

Validation

Static and focused validation should include:

cargo +nightly fmt --all -- --check
git diff --check
cargo clippy --workspace --all-targets --locked -- -D warnings
make lint
make test
RUSTDOCFLAGS="-D warnings" cargo doc --workspace --no-deps

Focused coverage includes:

• optional selection-policy parsing;
• strict rejection of unknown selection-policy fields;
• supported and unsupported selection modes;
• deterministic grouping under input permutations;
• capability-scoped group numbering;
• admission-boundary separation;
• locality-boundary separation;
• freshness-boundary separation;
• score differences not splitting equivalent groups;
• selection policy and groups affecting semantic digests;
• overlay-sync preservation of additive candidate fields;
• simultaneous Helm rendering of admission and selection policies;
• SWIM seed rendering;
• image-override propagation;
• provider-traffic topology materialization;
• token-rate-limit topology materialization.

Runtime provider-selection validation covers:

• one consumer gateway;
• three provider gateways;
• global Grid convergence;
• explicit roundRobin overlay mode;
• stable overlay revision;
• successful provider attribution;
• equal request distribution;
• no request-time control-plane access;
• clean teardown.

The distributed quota validation is maintained separately and covers shared quota across consumer gateway replicas while Grid continues to distribute admitted traffic.

Checklist

• I reviewed every changed line and can explain the change.
• API and overlay changes are documented.
• CRD and Helm configuration are included.
• New contracts include focused tests.
• Semantic digest behavior is covered.
• Request-path ownership remains in Praxis AI.
• Provider-selection behavior has functional demo coverage.
• Personal image references are excluded from repository defaults.
• Commits are signed and include a Signed-off-by trailer.

Breaking Changes

No breaking change is intended.

selectionPolicy and selection_group are additive. Existing Grid resources may omit the policy, and compatible consumers retain deterministic selection.

Deployments must upgrade Praxis AI to a version that understands the selection contract before enabling a non-deterministic selection mode. Older consumers may ignore additive candidate fields, but operators should not assume they will perform round-robin or random selection.

Quota enforcement remains an independent Praxis AI capability and is not enabled by this Grid change.

[praxis-bot]

PR Review

Summary: Adds provider-selection contract (selectionPolicy.mode) to GridNetwork CRD with deterministic, roundRobin, and random modes; adds selection-group assignment to overlay candidates; includes overlay-sync validation, Helm/CRD support, and a distributed token-quota Forge topology.

Overall: The design is well-structured with clear separation between operator-side rendering and data-plane selection. Tests cover round-trip serialization, unknown-field rejection, digest semantics, and group assignment. Two issues found: one indentation bug in the consumer config renderer that produces malformed YAML, and one forward-compatibility concern in overlay-sync.

Severity	Count
Critical	0
Large	1
Medium	1

[praxis-bot]

[Large] YAML indentation mismatch: selection_policy: is rendered with 9 leading spaces, but its sibling properties (local_site:, model_header:, candidates:) all use 8 spaces (via \x20 + 7 literal spaces in the format string at line 172-175). This 1-space offset will produce malformed YAML when selection_policy is present -- the YAML parser will reject the config or misinterpret the structure because selection_policy: appears at a different indentation level than its mapping siblings.

Change to 8 spaces for selection_policy: and 10 for mode: to match the existing \x20 + 7-space pattern:

format!("        selection_policy:\n          mode: {mode}\n")

The test at line 674 only checks config.contains("selection_policy:") which passes regardless of indentation -- consider adding a YAML parse assertion.

[praxis-bot]

[Medium] deny_unknown_fields on SelectionPolicy creates a forward-compatibility asymmetry with RoutingCandidate (which intentionally accepts unknown fields, tested by additive_candidate_field_is_digest_significant_and_accepted). If a future operator version adds a field to SelectionPolicyConfig, overlay-sync with the current code will reject the envelope as Malformed, requiring overlay-sync to be upgraded before the operator.

Either remove deny_unknown_fields from SelectionPolicy to match the open candidate contract, or document in a code comment that this struct is intentionally strict and that overlay-sync must be upgraded before the operator when new selection-policy fields are added.