Skip to content

Converge chat and triage on one Claude credential, and stop trusting the sender login unsigned - #163

Merged
imaustink merged 6 commits into
mainfrom
fix/converge-chat-triage-credentials
Jul 26, 2026
Merged

Converge chat and triage on one Claude credential, and stop trusting the sender login unsigned#163
imaustink merged 6 commits into
mainfrom
fix/converge-chat-triage-credentials

Conversation

@imaustink

@imaustink imaustink commented Jul 26, 2026

Copy link
Copy Markdown
Owner

Applying the ai-triage label on #161 asked me to authorize Claude, while chat had been launching agents all day on the same account. The orchestrator's own logs, fifteen minutes apart:

15:57 [authorization] {"verdict":"authorized",   "agentId":"claude-code-swe-agent", "actorLogin":null}
16:12 [authorization] {"verdict":"link-required","agentId":"claude-code-swe-agent", "pending":["claude@github:imaustink"]}

ADR 0029's cross-entry-point sharing was working in one direction only. A webhook turn always carries a verified senderLogin, so it always resolved a principal and read claude@github:imaustink. Chat could only read a login off an existing github link — and one of those was only ever created as a side effect of an Agent's identityProviders, which ADR 0030 §5 removed github from to fix a production 401. So the fix for the 401 silently switched off sharing for every chat caller. Neither flow was broken alone; they never met, and no amount of authorizing in one fixed the other.

What changes

Establishing a principal is its own pre-flight step (ADR 0031), independent of what the Agent declares: an ordinary github link that is link-only — its token is never injected, so no GITHUB_TOKEN reaches the run and the agent's delegated-write path (the observed 401) stays unreachable. Mapping and credential provisioning are now separately reachable, which is what §5 asked for and had no mechanism for.

Existing credentials are moved, not re-authorized. Both flows reading one key doesn't put anything at that key — records written before principals existed sit under the entry point's own subject. The gateway gains ClaudeTokenStore.rekey + POST /claude-auth/api/rekey, and the pre-flight adopts lazily on the turn that needs it. So converging costs a chat user one GitHub link and zero re-authorization.

The unsigned sender login is closed. Neither AGENT_SENDER_ASSERTION_SECRET nor GATEWAY_SENDER_ASSERTION_SECRET exists on the live deployment, so the orchestrator trusts the sender login from the /invoke body, unsigned — and that field selects whose stored credentials a run gets. The mechanism has existed since ADR 0030; what was missing is that these Secrets are hand-created and nothing said to include the key. e2e's bootstrap already sets it, so minikube ran the signed path while production ran the unsigned one.

Security properties, each pinned by a test

  • Identity.perUser — newly asserted by the one resolver that knows it structurally (OpenWebUiForwardedUserResolver) — gates both establishing a principal and adopting a credential. A login filed under the gateway's shared service subject would be inherited by every later senderLogin-less webhook turn, handing one person's credentials to everyone. progressListener ("has a live channel") was tried as the signal and rejected: a shared subject on a streaming caller passes it, so it's unsound in the one direction that leaks.
  • A pending principal link stops the turn rather than filing the credentials the user is about to create under a subject they're one link away from abandoning. A deliberate exception to §4's batching, which assumes providers are independent.
  • Rekey moves, never copies (the claude-remote write-back only writes the new key, so a copy rots into "Login expired"), never overwrites the destination, and deletes the source only after reading the destination backset swallows its own Redis errors, so an unverified delete could drop someone's only credential.
  • Provider order stays irrelevant (§4), and every failure degrades to the raw subject rather than blocking: sharing is an improvement, not a precondition.
  • The authorized verdict now carries its principal and delegateToAgent adopts it — otherwise the expired-credential path invalidates a record that was never written and tells the user to retry, the infinite loop that path exists to prevent.

e2e now drives both entry points

The suite existed to catch keying bugs and couldn't see this one, structurally: every spec drove webhooks, so the flow that couldn't resolve a principal was unreachable from a test. e2e/support/chat.ts mints the per-user X-OpenWebUI-User-Jwt and posts a streaming /v1/chat/completions; chat-harness.e2e.ts checks that minting against the real resolver and needs no cluster, so harness/product drift fails fast instead of surfacing as "could not resolve caller identity" inside every chat spec.

The rule worth keeping, now in e2e/README.md: when behaviour differs per entry point, cover it from each of them. Every keying bug in this repo's history has been an asymmetry between the two.

Review notes / limits

  • The prod Secret is not patched by this PR. Run ./scripts/provision-sender-assertion-secret.sh — one generated value, both Secrets, digest-compared before anything rolls, then gateway first, orchestrator second. That order is load-bearing: gateway-set/orchestrator-unset is exactly today's behaviour, while the reverse makes every webhook turn lose its sender login and re-prompt.

  • Run on minikube: all 11 pass (4 webhook + 4 chat + 3 harness, 12.5 min), against images built from this branch. The rekey route was also smoke-tested against the live gateway store: 200 {"status":"not-found"}, 400 on malformed input, unknown actions still 404. The orchestrator's own logs show a chat turn resolving principal: github:e2e-user — the key a triage turn reads — and openwebui:e2e-chat-user in the no-link case.

    The run found two defects, both in my harness rather than the product, fixed in f9f6177 + 2303fde:

    • The planner preferred opencode-swe-agent over the agent the request named. It declares no identityProviders, so the pre-flight iterated an empty list and launched with no credential resolved — a silently-passing spec, not a weaker one. Pointing the triage route away from it sufficed while every spec arrived through an IntegrationRoute; chat has no route, so it is now disabled in the e2e overlay entirely.
    • chatTurn treated a parked turn as a failure, when parking is the product working (a turn needing a link holds open for the flow expiry). Now explicit via allowPark. Two attempts to get right: the abort fires on the body read, not fetch (SSE headers flush immediately), and AbortSignal.timeout rejects with a DOMException that fails instanceof Error — either alone made the option a no-op. Also cut that spec from 845s to 93s.
  • The GitHub link is seeded in e2e, not established — a real device/authcode round trip needs github.com and a browser. The orchestrator reads only githubLogin off that record, so the path under test is identical; the OAuth exchange itself stays uncovered, as before.

  • Not fixed here: resolveActorLogin still trusts a github link under whatever subject it finds, including a shared one, if a leftover record exists there (the identity-link store has no TTL). This change never creates one there. Removing that hazard belongs with ADR 0030 §6's alias table.

  • Commits are ordered so each builds and tests on its own: rekey capability → convergence that uses it → e2e → prod bootstrap.

Verified: all workspaces green (451 orchestrator, 236 gateway, 3 harness), typecheck clean at each commit, helm template renders for both e2e and production values.

🤖 Generated with Claude Code

https://claude.ai/code/session_01W7XfNb5HsmaZH7QyxUrbyB

imaustink and others added 6 commits July 25, 2026 17:09
Re-keying where a credential is READ from does not move the credential.
Records written before principals existed (ADR 0029/0030 §6) sit under the
entry point's own subject, so converging the two entry points would still
have charged every existing user a fresh login to reproduce a credential
the gateway is holding the whole time.

Adds the capability that makes that unnecessary: `ClaudeTokenStore.rekey`
and a bearer-gated `POST /claude-auth/api/rekey`, plus `IdentityLinkPort.rekey`
and both claude clients' side of it. No caller yet -- the pre-flight change
that uses it is the next commit.

Deliberate properties, each with a test:

- MOVE, not copy. The claude-remote write-back only ever writes the new key,
  so a leftover copy would rot and then fail whichever flow still read it
  with "Login expired" -- worse than either alternative.
- Never overwrites the destination: a record already there is by definition
  at least as current as the one being moved.
- Deletes the source only after reading the destination back. `set` swallows
  its own Redis errors by design, so an unverified delete could drop a
  human's only credential on a transient failure.
- A self-move is a no-op rather than a delete-what-was-just-written.
- Bearer-gated like `token` and `invalidate`, which already hand out and
  destroy any subject's credential -- it grants no new authority over the
  keyspace. Authorizing that both subjects are the same human is the
  caller's job, and the route says so.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01W7XfNb5HsmaZH7QyxUrbyB
ADR 0029's cross-entry-point sharing worked in one direction only, which left
the reported bug fully intact for the flow it was reported from:

  15:57 [authorization] {"verdict":"authorized",   "actorLogin":null}
  16:12 [authorization] {"verdict":"link-required","pending":["claude@github:imaustink"]}

Same human, fifteen minutes apart, chat working and ai-triage prompting. A
webhook turn always carries a verified `senderLogin`, so it always resolved a
principal; chat could only read a login off an existing `github` link, and one
of those was only ever created as a side effect of an Agent's
`identityProviders` -- which ADR 0030 §5 removed `github` from to fix a
production 401. So the fix for the 401 silently switched off sharing for every
chat caller. Neither flow was broken alone; they simply never met.

Establishing a principal becomes its own pre-flight step (ADR 0031),
independent of what the Agent declares: an ordinary `github` link that is
LINK-ONLY -- its token is never injected, so no GITHUB_TOKEN reaches the run
and the agent's delegated-write path stays unreachable. Mapping and credential
provisioning are now separately reachable, which is what §5 asked for and had
no mechanism for.

Safety and ordering, each pinned by a test:

- Gated on `Identity.perUser`, newly asserted by the one resolver that knows
  it structurally (`OpenWebUiForwardedUserResolver`). A login filed under the
  gateway's shared service subject would be inherited by every later
  senderLogin-less webhook turn, handing one person's credentials to everyone.
  A live channel (`progressListener`) was tried as the signal and rejected: a
  shared subject on a streaming caller passes it, so it is unsound in the one
  direction that leaks.
- The principal step runs first, so CRD provider order stays irrelevant (§4).
- A pending principal link stops the turn rather than filing the credentials
  the user is about to create under a subject they are one link away from
  abandoning. A deliberate exception to §4's batching, which assumes the
  providers are independent.
- Every failure degrades to the raw subject rather than blocking: sharing is
  an improvement, not a precondition.

Existing credentials are ADOPTED rather than re-authorized, using the rekey
added in the previous commit -- lazily, on the turn that needs it, because the
(subject -> principal) mapping only exists on a caller's own authenticated
turn. Same `perUser` gate, for the same reason.

The `authorized` verdict now carries the `principal` its credentials were
keyed by and `delegateToAgent` adopts it: otherwise the expired-credential
path re-derives the pre-upgrade key, invalidates a record that was never
written, and tells the user to retry -- the infinite loop that path exists to
prevent.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01W7XfNb5HsmaZH7QyxUrbyB
The suite existed specifically to catch credential-keying bugs and could not
see this one, for a structural reason: every spec drove GitHub webhooks. The
webhook path always resolves a principal, so a convergence that worked in only
that direction passed every spec while the chat path -- the one that could not
resolve a principal -- was unreachable from a test.

`support/chat.ts` closes that: it mints the per-request JWT Open WebUI signs
(`X-OpenWebUI-User-Jwt`) and posts a STREAMING `/v1/chat/completions`, which is
what makes the caller per-user and gives the turn the live channel the real
chat surface has. Hand-rolled HS256 rather than adding `jose` -- e2e/ has kept
its dependencies to vitest + typescript, same precedent as sender-assertion.ts.

`chat-harness.e2e.ts` checks the harness against the REAL resolver and needs no
cluster, so it runs anywhere. Signing is the part of a harness most likely to be
quietly wrong, and when it is, every cluster spec built on it fails with "could
not resolve caller identity" -- which reads as a product defect and costs a full
stack cycle to disprove.

Four chat-side cases in identity-keying.e2e.ts: resolves from the principal (and
asserts no GITHUB_TOKEN reaches the run), adopts a pre-principal record without
re-prompting (asserting the old key is GONE, not copied), refuses another
human's credential, and still works with no GitHub link at all.

The GitHub link is seeded (`seedGithubLink`), not established: a real
device/authcode round trip needs github.com and a human with a browser. The
orchestrator reads only `githubLogin` off that record, so the path under test is
identical -- what stays uncovered is the OAuth exchange, as before.

values-e2e.yaml now configures the forwarded-JWT secret, without which every
e2e chat caller collapses onto the shared service subject and the specs would
silently exercise the degraded path.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01W7XfNb5HsmaZH7QyxUrbyB
Observed on the live deployment: neither AGENT_SENDER_ASSERTION_SECRET nor
GATEWAY_SENDER_ASSERTION_SECRET is present, so both processes log their startup
warning and the orchestrator trusts the sender login from the /invoke request
BODY, unsigned. That field selects the caller's principal and therefore which
stored Claude credentials a run receives (ADR 0030 §6, 0031) -- so anything
holding the gateway's /invoke token can name a login and be handed that
person's credentials.

The mechanism has existed since ADR 0030; what was missing is that these
Secrets are hand-created and nothing told anyone to include the key. e2e's
bootstrap already sets it on both sides, so minikube was exercising the signed
path while production ran the unsigned one.

Adds it to values-production.yaml and README's bootstrap (with why, and the
matching-value requirement), plus a script that performs the change on a live
deployment: one generated value, both Secrets, digest-compared before anything
rolls, then gateway first and orchestrator second.

That order is load-bearing. Gateway-set/orchestrator-unset is exactly today's
behaviour (the gateway signs, the orchestrator ignores the header and keeps
reading the body field). The reverse makes the orchestrator require a signature
nobody is producing, so every webhook turn loses its sender login, falls back to
the shared service subject, and re-prompts for authorization.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01W7XfNb5HsmaZH7QyxUrbyB
A webhook turn reaches its agent through an IntegrationRoute; chat has no such
thing. `/v1/chat/completions` carries no event descriptor, and a session's
`activeAgentId` only CONTINUES an existing run rather than launching one, so
selection goes through the planner -- which the rest of the suite deliberately
never depends on.

Two fixes, since the surface offers no deterministic hook:

- The request names the target agent outright, which is as close to
  deterministic as chat gets.
- Every launch assertion goes through `expectCredentialAgent`, which asserts the
  launched `spec.agentRef` is one that actually declares the credentials under
  test, and FAILS NAMING what the planner picked instead. The e2e catalog holds
  three agents; two declare `claude`/`claude-remote` and either is a valid
  outcome for what these specs test (which SUBJECT a credential came from), while
  `opencode-swe-agent` declares none -- picking it previously surfaced as a
  confusing complaint about missing env vars, or as a 420s timeout with nothing
  saying which agent ran.

The non-determinism is now stated in the spec's own doc rather than left for the
next person to discover from a flake.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01W7XfNb5HsmaZH7QyxUrbyB
Neither was visible to typecheck, and both made a chat spec fail (or worse,
pass) for reasons that had nothing to do with the code under test.

1. `opencode-swe-agent` was in the e2e catalog, and the planner preferred it
   over the agent the request named -- its description is the closest match for
   any "fix the failing test in a repo" text. It declares no identityProviders,
   so the pre-flight iterates an empty list and launches with no credential
   resolved at all. Pointing the triage ROUTE away from it was enough while
   every spec arrived through an IntegrationRoute; a chat turn has no route.
   That is not a weaker version of the test but a different, silently-passing
   one, so the agent is now disabled in the e2e overlay entirely: an agent that
   cannot exercise the behaviour under test does not belong in the catalog the
   tests select from.

2. `chatTurn` treated a parked turn as a failure. A turn that needs an account
   link and has a live channel deliberately holds its connection open for the
   whole flow expiry waiting for the human -- so for the negative control, whose
   expected outcome is "no launch", parking IS the product working. It now
   returns `undefined` under an explicit `allowPark`, leaving a genuine hang in
   any other spec still failing loudly.

   Getting that right took two goes, both worth recording: the abort fires on
   the BODY READ, not on `fetch` (the orchestrator flushes SSE headers
   immediately, so `fetch` resolves as soon as they land), and
   `AbortSignal.timeout` rejects with a DOMException that fails
   `instanceof Error`. Either alone made `allowPark` a silent no-op.

The park window also doubles as the "give it time to launch if it were going to"
wait, which cut that spec from 845s to 93s.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01W7XfNb5HsmaZH7QyxUrbyB
@imaustink

Copy link
Copy Markdown
Owner Author

Ran the suite on minikube against images built from this branch: 11/11 pass (4 webhook + 4 chat + 3 harness, 12.5 min).

Two defects surfaced, both in the harness rather than the product — recorded because each was invisible to typecheck and one of them passes silently:

  1. The planner preferred opencode-swe-agent over the agent the request named. It declares no identityProviders, so the authorization pre-flight iterated an empty list and launched with no credential resolved at all. Every weaker assertion in the spec still passed. Pointing the triage route away from that agent was enough while every spec arrived through an IntegrationRoute — a chat turn has no route, so it is now out of the e2e catalog entirely.
  2. chatTurn treated a parked turn as a failure. A turn that needs an account link and has a live channel deliberately holds its connection open for the whole flow expiry, so for the negative control parking is the correct outcome. Now explicit via allowPark, leaving a genuine hang failing loudly elsewhere.

Also smoke-tested the new gateway route against the live store: 200 {"status":"not-found"}, 400 on malformed input, and unknown actions still 404 (so it is genuinely registered, not swallowed by a catch-all).

Body updated — the "not executed" caveat is gone.

@imaustink
imaustink merged commit 907d7d3 into main Jul 26, 2026
1 check passed
@imaustink
imaustink deleted the fix/converge-chat-triage-credentials branch July 26, 2026 02:02
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant