Converge chat and triage on one Claude credential, and stop trusting the sender login unsigned - #163
Merged
Merged
Conversation
Re-keying where a credential is READ from does not move the credential. Records written before principals existed (ADR 0029/0030 §6) sit under the entry point's own subject, so converging the two entry points would still have charged every existing user a fresh login to reproduce a credential the gateway is holding the whole time. Adds the capability that makes that unnecessary: `ClaudeTokenStore.rekey` and a bearer-gated `POST /claude-auth/api/rekey`, plus `IdentityLinkPort.rekey` and both claude clients' side of it. No caller yet -- the pre-flight change that uses it is the next commit. Deliberate properties, each with a test: - MOVE, not copy. The claude-remote write-back only ever writes the new key, so a leftover copy would rot and then fail whichever flow still read it with "Login expired" -- worse than either alternative. - Never overwrites the destination: a record already there is by definition at least as current as the one being moved. - Deletes the source only after reading the destination back. `set` swallows its own Redis errors by design, so an unverified delete could drop a human's only credential on a transient failure. - A self-move is a no-op rather than a delete-what-was-just-written. - Bearer-gated like `token` and `invalidate`, which already hand out and destroy any subject's credential -- it grants no new authority over the keyspace. Authorizing that both subjects are the same human is the caller's job, and the route says so. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01W7XfNb5HsmaZH7QyxUrbyB
ADR 0029's cross-entry-point sharing worked in one direction only, which left
the reported bug fully intact for the flow it was reported from:
15:57 [authorization] {"verdict":"authorized", "actorLogin":null}
16:12 [authorization] {"verdict":"link-required","pending":["claude@github:imaustink"]}
Same human, fifteen minutes apart, chat working and ai-triage prompting. A
webhook turn always carries a verified `senderLogin`, so it always resolved a
principal; chat could only read a login off an existing `github` link, and one
of those was only ever created as a side effect of an Agent's
`identityProviders` -- which ADR 0030 §5 removed `github` from to fix a
production 401. So the fix for the 401 silently switched off sharing for every
chat caller. Neither flow was broken alone; they simply never met.
Establishing a principal becomes its own pre-flight step (ADR 0031),
independent of what the Agent declares: an ordinary `github` link that is
LINK-ONLY -- its token is never injected, so no GITHUB_TOKEN reaches the run
and the agent's delegated-write path stays unreachable. Mapping and credential
provisioning are now separately reachable, which is what §5 asked for and had
no mechanism for.
Safety and ordering, each pinned by a test:
- Gated on `Identity.perUser`, newly asserted by the one resolver that knows
it structurally (`OpenWebUiForwardedUserResolver`). A login filed under the
gateway's shared service subject would be inherited by every later
senderLogin-less webhook turn, handing one person's credentials to everyone.
A live channel (`progressListener`) was tried as the signal and rejected: a
shared subject on a streaming caller passes it, so it is unsound in the one
direction that leaks.
- The principal step runs first, so CRD provider order stays irrelevant (§4).
- A pending principal link stops the turn rather than filing the credentials
the user is about to create under a subject they are one link away from
abandoning. A deliberate exception to §4's batching, which assumes the
providers are independent.
- Every failure degrades to the raw subject rather than blocking: sharing is
an improvement, not a precondition.
Existing credentials are ADOPTED rather than re-authorized, using the rekey
added in the previous commit -- lazily, on the turn that needs it, because the
(subject -> principal) mapping only exists on a caller's own authenticated
turn. Same `perUser` gate, for the same reason.
The `authorized` verdict now carries the `principal` its credentials were
keyed by and `delegateToAgent` adopts it: otherwise the expired-credential
path re-derives the pre-upgrade key, invalidates a record that was never
written, and tells the user to retry -- the infinite loop that path exists to
prevent.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01W7XfNb5HsmaZH7QyxUrbyB
The suite existed specifically to catch credential-keying bugs and could not see this one, for a structural reason: every spec drove GitHub webhooks. The webhook path always resolves a principal, so a convergence that worked in only that direction passed every spec while the chat path -- the one that could not resolve a principal -- was unreachable from a test. `support/chat.ts` closes that: it mints the per-request JWT Open WebUI signs (`X-OpenWebUI-User-Jwt`) and posts a STREAMING `/v1/chat/completions`, which is what makes the caller per-user and gives the turn the live channel the real chat surface has. Hand-rolled HS256 rather than adding `jose` -- e2e/ has kept its dependencies to vitest + typescript, same precedent as sender-assertion.ts. `chat-harness.e2e.ts` checks the harness against the REAL resolver and needs no cluster, so it runs anywhere. Signing is the part of a harness most likely to be quietly wrong, and when it is, every cluster spec built on it fails with "could not resolve caller identity" -- which reads as a product defect and costs a full stack cycle to disprove. Four chat-side cases in identity-keying.e2e.ts: resolves from the principal (and asserts no GITHUB_TOKEN reaches the run), adopts a pre-principal record without re-prompting (asserting the old key is GONE, not copied), refuses another human's credential, and still works with no GitHub link at all. The GitHub link is seeded (`seedGithubLink`), not established: a real device/authcode round trip needs github.com and a human with a browser. The orchestrator reads only `githubLogin` off that record, so the path under test is identical -- what stays uncovered is the OAuth exchange, as before. values-e2e.yaml now configures the forwarded-JWT secret, without which every e2e chat caller collapses onto the shared service subject and the specs would silently exercise the degraded path. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01W7XfNb5HsmaZH7QyxUrbyB
Observed on the live deployment: neither AGENT_SENDER_ASSERTION_SECRET nor GATEWAY_SENDER_ASSERTION_SECRET is present, so both processes log their startup warning and the orchestrator trusts the sender login from the /invoke request BODY, unsigned. That field selects the caller's principal and therefore which stored Claude credentials a run receives (ADR 0030 §6, 0031) -- so anything holding the gateway's /invoke token can name a login and be handed that person's credentials. The mechanism has existed since ADR 0030; what was missing is that these Secrets are hand-created and nothing told anyone to include the key. e2e's bootstrap already sets it on both sides, so minikube was exercising the signed path while production ran the unsigned one. Adds it to values-production.yaml and README's bootstrap (with why, and the matching-value requirement), plus a script that performs the change on a live deployment: one generated value, both Secrets, digest-compared before anything rolls, then gateway first and orchestrator second. That order is load-bearing. Gateway-set/orchestrator-unset is exactly today's behaviour (the gateway signs, the orchestrator ignores the header and keeps reading the body field). The reverse makes the orchestrator require a signature nobody is producing, so every webhook turn loses its sender login, falls back to the shared service subject, and re-prompts for authorization. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01W7XfNb5HsmaZH7QyxUrbyB
A webhook turn reaches its agent through an IntegrationRoute; chat has no such thing. `/v1/chat/completions` carries no event descriptor, and a session's `activeAgentId` only CONTINUES an existing run rather than launching one, so selection goes through the planner -- which the rest of the suite deliberately never depends on. Two fixes, since the surface offers no deterministic hook: - The request names the target agent outright, which is as close to deterministic as chat gets. - Every launch assertion goes through `expectCredentialAgent`, which asserts the launched `spec.agentRef` is one that actually declares the credentials under test, and FAILS NAMING what the planner picked instead. The e2e catalog holds three agents; two declare `claude`/`claude-remote` and either is a valid outcome for what these specs test (which SUBJECT a credential came from), while `opencode-swe-agent` declares none -- picking it previously surfaced as a confusing complaint about missing env vars, or as a 420s timeout with nothing saying which agent ran. The non-determinism is now stated in the spec's own doc rather than left for the next person to discover from a flake. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01W7XfNb5HsmaZH7QyxUrbyB
Neither was visible to typecheck, and both made a chat spec fail (or worse, pass) for reasons that had nothing to do with the code under test. 1. `opencode-swe-agent` was in the e2e catalog, and the planner preferred it over the agent the request named -- its description is the closest match for any "fix the failing test in a repo" text. It declares no identityProviders, so the pre-flight iterates an empty list and launches with no credential resolved at all. Pointing the triage ROUTE away from it was enough while every spec arrived through an IntegrationRoute; a chat turn has no route. That is not a weaker version of the test but a different, silently-passing one, so the agent is now disabled in the e2e overlay entirely: an agent that cannot exercise the behaviour under test does not belong in the catalog the tests select from. 2. `chatTurn` treated a parked turn as a failure. A turn that needs an account link and has a live channel deliberately holds its connection open for the whole flow expiry waiting for the human -- so for the negative control, whose expected outcome is "no launch", parking IS the product working. It now returns `undefined` under an explicit `allowPark`, leaving a genuine hang in any other spec still failing loudly. Getting that right took two goes, both worth recording: the abort fires on the BODY READ, not on `fetch` (the orchestrator flushes SSE headers immediately, so `fetch` resolves as soon as they land), and `AbortSignal.timeout` rejects with a DOMException that fails `instanceof Error`. Either alone made `allowPark` a silent no-op. The park window also doubles as the "give it time to launch if it were going to" wait, which cut that spec from 845s to 93s. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01W7XfNb5HsmaZH7QyxUrbyB
Owner
Author
|
Ran the suite on minikube against images built from this branch: 11/11 pass (4 webhook + 4 chat + 3 harness, 12.5 min). Two defects surfaced, both in the harness rather than the product — recorded because each was invisible to typecheck and one of them passes silently:
Also smoke-tested the new gateway route against the live store: Body updated — the "not executed" caveat is gone. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Applying the
ai-triagelabel on #161 asked me to authorize Claude, while chat had been launching agents all day on the same account. The orchestrator's own logs, fifteen minutes apart:ADR 0029's cross-entry-point sharing was working in one direction only. A webhook turn always carries a verified
senderLogin, so it always resolved a principal and readclaude@github:imaustink. Chat could only read a login off an existinggithublink — and one of those was only ever created as a side effect of an Agent'sidentityProviders, which ADR 0030 §5 removedgithubfrom to fix a production 401. So the fix for the 401 silently switched off sharing for every chat caller. Neither flow was broken alone; they never met, and no amount of authorizing in one fixed the other.What changes
Establishing a principal is its own pre-flight step (ADR 0031), independent of what the Agent declares: an ordinary
githublink that is link-only — its token is never injected, so noGITHUB_TOKENreaches the run and the agent's delegated-write path (the observed 401) stays unreachable. Mapping and credential provisioning are now separately reachable, which is what §5 asked for and had no mechanism for.Existing credentials are moved, not re-authorized. Both flows reading one key doesn't put anything at that key — records written before principals existed sit under the entry point's own subject. The gateway gains
ClaudeTokenStore.rekey+POST /claude-auth/api/rekey, and the pre-flight adopts lazily on the turn that needs it. So converging costs a chat user one GitHub link and zero re-authorization.The unsigned sender login is closed. Neither
AGENT_SENDER_ASSERTION_SECRETnorGATEWAY_SENDER_ASSERTION_SECRETexists on the live deployment, so the orchestrator trusts the sender login from the/invokebody, unsigned — and that field selects whose stored credentials a run gets. The mechanism has existed since ADR 0030; what was missing is that these Secrets are hand-created and nothing said to include the key. e2e's bootstrap already sets it, so minikube ran the signed path while production ran the unsigned one.Security properties, each pinned by a test
Identity.perUser— newly asserted by the one resolver that knows it structurally (OpenWebUiForwardedUserResolver) — gates both establishing a principal and adopting a credential. A login filed under the gateway's shared service subject would be inherited by every latersenderLogin-less webhook turn, handing one person's credentials to everyone.progressListener("has a live channel") was tried as the signal and rejected: a shared subject on a streaming caller passes it, so it's unsound in the one direction that leaks.claude-remotewrite-back only writes the new key, so a copy rots into "Login expired"), never overwrites the destination, and deletes the source only after reading the destination back —setswallows its own Redis errors, so an unverified delete could drop someone's only credential.authorizedverdict now carries itsprincipalanddelegateToAgentadopts it — otherwise the expired-credential path invalidates a record that was never written and tells the user to retry, the infinite loop that path exists to prevent.e2e now drives both entry points
The suite existed to catch keying bugs and couldn't see this one, structurally: every spec drove webhooks, so the flow that couldn't resolve a principal was unreachable from a test.
e2e/support/chat.tsmints the per-userX-OpenWebUI-User-Jwtand posts a streaming/v1/chat/completions;chat-harness.e2e.tschecks that minting against the real resolver and needs no cluster, so harness/product drift fails fast instead of surfacing as "could not resolve caller identity" inside every chat spec.The rule worth keeping, now in
e2e/README.md: when behaviour differs per entry point, cover it from each of them. Every keying bug in this repo's history has been an asymmetry between the two.Review notes / limits
The prod Secret is not patched by this PR. Run
./scripts/provision-sender-assertion-secret.sh— one generated value, both Secrets, digest-compared before anything rolls, then gateway first, orchestrator second. That order is load-bearing: gateway-set/orchestrator-unset is exactly today's behaviour, while the reverse makes every webhook turn lose its sender login and re-prompt.Run on minikube: all 11 pass (4 webhook + 4 chat + 3 harness, 12.5 min), against images built from this branch. The
rekeyroute was also smoke-tested against the live gateway store:200 {"status":"not-found"},400on malformed input, unknown actions still404. The orchestrator's own logs show a chat turn resolvingprincipal: github:e2e-user— the key a triage turn reads — andopenwebui:e2e-chat-userin the no-link case.The run found two defects, both in my harness rather than the product, fixed in f9f6177 + 2303fde:
opencode-swe-agentover the agent the request named. It declares noidentityProviders, so the pre-flight iterated an empty list and launched with no credential resolved — a silently-passing spec, not a weaker one. Pointing the triage route away from it sufficed while every spec arrived through anIntegrationRoute; chat has no route, so it is now disabled in the e2e overlay entirely.chatTurntreated a parked turn as a failure, when parking is the product working (a turn needing a link holds open for the flow expiry). Now explicit viaallowPark. Two attempts to get right: the abort fires on the body read, notfetch(SSE headers flush immediately), andAbortSignal.timeoutrejects with aDOMExceptionthat failsinstanceof Error— either alone made the option a no-op. Also cut that spec from 845s to 93s.The GitHub link is seeded in e2e, not established — a real device/authcode round trip needs github.com and a browser. The orchestrator reads only
githubLoginoff that record, so the path under test is identical; the OAuth exchange itself stays uncovered, as before.Not fixed here:
resolveActorLoginstill trusts agithublink under whatever subject it finds, including a shared one, if a leftover record exists there (the identity-link store has no TTL). This change never creates one there. Removing that hazard belongs with ADR 0030 §6's alias table.Commits are ordered so each builds and tests on its own: rekey capability → convergence that uses it → e2e → prod bootstrap.
Verified: all workspaces green (451 orchestrator, 236 gateway, 3 harness), typecheck clean at each commit,
helm templaterenders for both e2e and production values.🤖 Generated with Claude Code
https://claude.ai/code/session_01W7XfNb5HsmaZH7QyxUrbyB