Add extensible chat client routing

[joshuajyue]

Summary

Adds experimental chat-routing primitives for selecting configured IChatClient instances without imposing a particular routing policy.

• RoutingChatClient supports one-shot client selection.
• FailoverChatClient retries uncanceled failures that occur before output is exposed.
• OrderedFailoverChatClient provides ordered fallback.
• SemanticRoutingChatClient selects clients using cached profile embeddings.
• RoutingContext exposes mutable request messages and options.
• FailoverChatClientAttempt reports invocation outcome, duration, time to first update, completion, and output commitment.

Related to #7647.

API shape

Routing policies have two seams:

protected override ValueTask<IChatClient> SelectClientAsync(
    RoutingContext context,
    CancellationToken cancellationToken);

protected override ValueTask OnRoutingUpdateAsync(
    RoutingContext context,
    FailoverChatClientAttempt? attempt,
    bool isTerminal,
    CancellationToken cancellationToken);

SelectClientAsync runs before every invocation. OnRoutingUpdateAsync reports each attempt so policy state can influence the next selection.

isTerminal means the base will not select another client after the callback completes. A null attempt represents selection terminating before a client was invoked.

Failover behavior

• Cancellation never triggers reselection.
• Streaming fallback occurs only before the first update is exposed.
• Mid-stream failures and caller abandonment are terminal.
• Enumerator disposal failures are included in the attempt.
• Attempt limits are request-local.
• Null selector results are treated as terminal selection failures.
• Ordered failover retains request state only between a nonterminal update and the immediately following selection; it holds no state while a provider invocation or stream is active.
• Streaming callers must dispose active enumerators for terminal observation and inner-resource cleanup.

Semantic routing

SemanticRoutingChatClient:

• Lazily embeds and caches profile utterances.
• Embeds the final user message for selection.
• Supports global top-K profile matching.
• Aggregates matches per client using mean or sum.
• Preserves the existing single-best-match behavior with topK: 1.
• Uses a required default client when no profile clears the threshold.
• Supports caller-controlled ownership through leaveOpen.

Tests

Coverage includes selection failures, retries, cancellation, attempt limits, streaming commitment, early disposal, enumerator failures, concurrent requests, state cleanup, semantic thresholds, top-K aggregation, caching, and ownership.

• 753 tests passed on net10.0
• net472 test project builds successfully

Microsoft Reviewers: Open in CodeFlow

[PranavSenthilnathan]

It seems a little unintuitive that context.ChatOptions is the input to SelectClientAsync but it's also the output since users can mutate it. If we expect people to return modified options, it should be part of the return type for SelectClientAsync. I also think the properties on RequestContext should be readonly for the same reason.

[joshuajyue]

I see what you mean. The official "API-standard" way ChatOptions should be configured is through a unique ConfiguredChatOptions client (this is better for failover, cooldowns, etc.). So, the user should not mutate ChatOptions inside SelectClientAsync. I'll go ahead and make RoutingContext get-only.

Making ChatOptions get-only does still allow the user to change ChatOptions, since it's a reference, but I'll put some documentation about how they should use ConfiguredChatOptions. I don't know if there is a viable way to absolutely block the user other than cloning.

If the user wants to, in the end, they can still mutate it, but they should know that'll be mutating the original caller-options.

[joshuajyue]

There is, however, a concern where certain RoutingChatClients, such as SemanticRoutingChatClient, rely on stable messages. Since IEnumerable<ChatMessage> may be lazy or one-shot, the selector and selected client could otherwise observe different results.

Originally, a derived RCC could stabilize messages by replacing them with an IReadOnlyList. Since messages is now get-only, I opted for an optional BufferMessages() method on RoutingContext. Routers only call it when needed; it reuses an existing IReadOnlyList, or enumerates and caches the sequence once for that request.

We could also leave this API out and leave repeatable enumeration up to the caller. This is the design addition I’m less certain about. I'll leave it in for now.

[jozkee]

I don't know if there is a viable way to absolutely block the user other than cloning.

Cloning is currently the only way.

[joshuajyue]

I'll go ahead and clone options so it's read-only / doesn't affect the caller options. Still leaving the conversation open however for the question about BufferMessages()

[joshuajyue]

After thinking through the ChatOptions more, I think supporting both configured clients and returned options is useful.

Not every option variation needs a distinct route identity. For example, gpt-5.5 low and high reasoning may belong to the same failure: if high fails, policy state keyed by that client should also apply to low. In that case, the IChatClient reference should remain the route identity while returned options shape only the current attempt.

When configurations do need independent policy state -- such as gpt-5.5 and gpt-5.6 being cooled down separately -- the user can select distinct ConfigureOptionsChatClient wrappers.

I am leaning toward a small per-attempt selection result rather than returning RoutingContext:

RoutingSelection(IChatClient client, ChatOptions? options)

RoutingContext would remain one stable object for the request, while each selection explicitly returns the client and options for that attempt. The contract would state that:

• IChatClient reference identity is the sole route identity.
• Returned options do not create a distinct route.
• OrderedFailoverChatClient continues to operate on unique client references.
• RoutingContext.ChatOptions is the request-level baseline shared across selections.
• RoutingSelection.ChatOptions is scoped to the current attempt.
• FailoverChatClientAttempt.ChatOptions records the options routing passed for that attempt.

[jozkee]

You don't need RoutingSelection to enable something like cooldown-all-clients-with-the-same-backend:

class SharedBackendFailoverChatClient(IReadOnlyList<(IChatClient Client, string BackendKey)> routes)
    : FailoverChatClient;

I'm not sure about ergonomic value; I personally like the simplicity of just returning an IChatClient and I think you made a good point in mentioning that ConfigureOptions can be used to describe different clients/routes.

[joshuajyue]

You don't need RoutingSelection to enable something like cooldown-all-clients-with-the-same-backend

I'm not sure about ergonomic value; I personally like the simplicity of just returning an IChatClient and I think you >made a good point in mentioning that ConfigureOptions can be used to describe different clients/routes.

Yeah, I agree. I stress-tested the RoutingSelection / ChatRoute directions, and they ended up adding more option semantics than they were worth. ChatRoute requires us to merge route options with caller options, while RoutingSelection introduces a second options path that mostly overlaps with ConfigureOptionsChatClient.

I kept SelectClientAsync returning IChatClient. RoutingContext.ChatOptions is now only a snapshot for selection, and each downstream failover attempt gets a fresh clone of the caller options so mutations from one client do not leak into the next attempt. If a policy needs a separate backend or failure-domain identity, it can own that alongside the client exactly like your (Client, BackendKey) example. Configurations that should have distinct client identity can still use separate ConfigureOptionsChatClient wrappers.

Addressed in 87d382d63.

[joshuajyue]

Addressed in commit 0abbfd1, along with some cancellation races

[jozkee]

I don't know if there is a viable way to absolutely block the user other than cloning.

Cloning is currently the only way.

[jozkee]

Looking good.

[joshuajyue]

Thank you David for the feedback, addressed in 8b79ae4