feat: add centralized inference gateway

[sam-saffron-jarvis]

What

Adds a centralized, multi-provider inference gateway while keeping satellites as full local term-llm runtimes.

A satellite retains its agent, prompt, sessions, tools, approvals, filesystem, jobs, UI, and normal provider:model selection. The gateway owns provider configuration, API credentials, OAuth/CLI homes, search/fetch credentials, policy, retries, and attributed usage.

Satellite configuration is reduced to a gateway connection plus explicit local exceptions:

gateway:
  url: http://inference-gateway:8090
  token_file: ~/.config/term-llm/gateway-token
  local_providers: [ollama]

Normal selection remains unchanged:

term-llm ask --provider chatgpt:gpt-5.6-sol-high '...'
term-llm chat --provider openrouter:moonshotai/kimi-k2

Gateway protocol

Adds a versioned private /g1 provider plane:

• authenticated provider/model catalog with ETags and stale-safe caching
• streaming provider-neutral requests/events
• cancellation and bounded retry budgets
• synchronous satellite-local tool callbacks, including inline CLI provider bridges
• sealed provider state bound to client and provider
• centralized search and fetch
• per-client provider/model policy, concurrency, and search/fetch limits
• client enrollment, revocation, rotation, and attributed usage

The gateway is a remote llm.Provider, not a remote agent. It does not run satellite sessions, skills, jobs, or local filesystem tools.

OpenAI Responses edge

term-llm gateway serve also exposes:

POST /v1/responses
GET  /v1/models

This lets Discourse and standard OpenAI clients use central providers directly with namespaced model IDs:

chatgpt/gpt-5.6-sol
openrouter/moonshotai/kimi-k2

The edge supports:

• streaming and non-streaming Responses
• text and reasoning summaries/encrypted replay metadata
• image and file data URLs
• flat Responses function tools and tool choice
• function calls, tool results, and stateless full-input continuation
• OpenAI-compatible usage and required event/document fields
• disconnect cancellation
• policy-filtered /v1/models
• sanitized non-200 failures before streaming output and standard failure events afterward

It deliberately does not instantiate a term-llm agent or execute client function calls on the gateway.

Discourse configuration:

Provider: OpenAI
URL:      https://gateway.example.com/v1/responses
API key:  <gateway client token>
Model:    chatgpt/gpt-5.6-sol

The implementation was matched against the current Discourse Responses endpoint, dialect, message processor, and specs under plugins/discourse-ai.

Security

• Unique high-entropy client credentials; hashes only at rest
• Immediate revocation observed by running gateways
• Single-use, TTL-bound, policy-bound enrollment
• Provider/model allow and deny policy
• Per-client inference concurrency and search/fetch limits
• Gateway state sealed with AES-GCM and bound to client/provider identity
• Satellite WorkingDir and local paths are structurally absent from the wire
• Gateway-created per-run directories are isolated, cleaned, and scavenged safely
• Provider errors are classified and sanitized; raw upstream bodies remain gateway-side
• CLI-backed providers are double-gated and denied by default
• Optional built-in TLS; private-network/reverse-proxy deployment documented

Operations

Adds:

term-llm gateway serve
term-llm gateway client add|list|revoke
term-llm gateway enroll
term-llm gateway usage

Enrollment writes a separate mode-0600 token file and updates satellite config atomically without printing the long-lived credential.

Includes a private-network Compose example, gateway/satellite example configs, health checks, CI YAML validation, and a full deployment/security guide.

Live testing

Tested with isolated gateway and satellite homes in throwaway directories:

• enrollment, config/token permissions, replay/expiry denial, and immediate revocation
• real gateway process and full satellite CLI routing
• provider/model listing and normal provider:model selection
• satellite-local tool execution with no gateway filesystem copy
• centralized search/fetch and attributed usage
• cancellation and bounded persistent-500 retries
• policy denial, bad credentials, unknown provider/model, local-provider exceptions
• /v1/models and streaming /v1/responses through curl, official openai-go, and a Discourse-shaped Ruby Net::HTTP parser

Matched direct and gateway debug:normal runs, six warm samples each:

Path	p50
Direct	3.4443s
Gateway	3.4411s
Difference	-3.2ms (-0.09%)

The difference is measurement noise; no measurable warm inference overhead was observed for that workload.

A 16-way gateway run completed 16/16 requests in 3.5019s wall time, with 3.4950s per-request p50 and 3.4988s maximum.

Verification

After rebasing onto da927604:

git diff --check upstream/main...HEAD
go build ./...
go test ./...
go vet ./...

All passed with isolated HOME/XDG directories.

Focused race coverage also passed for gateway, Responses, catalog, routing, tool callbacks, limits, retries, cancellation, enrollment, search, and usage.

Scope

Included: text inference, vision/file inputs, function tools/results, reasoning, search/fetch, usage, private satellite protocol, and OpenAI Responses edge.

Not included: image/audio generation, realtime/WebRTC proxying, multi-gateway HA, or provider-hosted non-function tools.

[sam-saffron-jarvis]

Follow-up: session binding and WebSocket parity

Audited and hardened multi-turn gateway resumption after the initial PR:

• sealed provider state is now cryptographically bound to gateway client, provider, and satellite session ID;
• added committed tests for multi-turn transcript reconstruction, tools/results, reasoning/provider replay, satellite/provider reconstruction, gateway restart, concurrent session isolation, and direct-vs-gateway request equivalence;
• added a 30-second gateway provider-session idle cache for sessionful /g1 requests using explicitly configured OpenAI/ChatGPT WebSocket providers;
• warm follow-ups reuse the same provider/WebSocket and therefore send previous_response_id plus only the continuation suffix, matching direct term-llm behavior;
• cancellation, failures, config/credential changes, idle expiry, and shutdown evict and clean cached providers;
• /v1/responses remains explicitly stateless.

Fake WebSocket test with an intentional 25ms handshake delay:

Request	Time
First turn	25.92ms
Warm follow-up	1.18ms
Follow-up after expiry	26.07ms

The first turn and warm follow-up used one provider and one WebSocket connection. After expiry, the gateway safely created a new connection and replayed the full transcript.