Make agent turns survive an orchestrator rollout, and unwedge terminal runs

[imaustink]

An agent turn that was mid-flight during an orchestrator rollout still failed, even after 55b0e4b. That commit fixed three real defects and made the failure honest; it could not make it stop happening. This does.

Bundles three smaller fixes the investigation surfaced, at @imaustink's request.

1. Resumable agent turns (docs/adr/0033)

An agent turn's work lives in its own Job pod. The turn's wait lives in an orchestrator pod. Every push to main restarts the orchestrator — the original incident recorded 11 rollouts in 14 hours — and the drain window is 25s against a turn that takes minutes. Past that, core NATS discards the reply published into the gap, because nothing is subscribed. The answer existed, was correct, and was unrecoverable.

JetStream would fix it at the transport, but the deployed NATS has no jetstream block, so that means new storage plus new retention/consumer failure modes. Instead this uses the pod that already outlives the orchestrator — the agent's own — as the buffer:

• The agent holds its concluding message until acked. New reply_ack down-message; publishHeld re-offers the identical envelope, seq included, every 10s until acked, giving up after 10min (AGENT_REPLY_ACK_TIMEOUT_MS, 0 disables). Questions are held too — a lost question strands a conversation as badly as a lost answer — and released as soon as their answer arrives. Narration isn't held.
• The conversation is anchored to the run before the wait. persistSession runs after the graph returns, which is precisely the write a SIGKILLed pod never performs.
• The next turn re-attaches. checkActiveAgentRun distinguishes "parked on a question" (publish the text as a prompt) from "owed a reply" (publish nothing, just collect), bounded at 45s since silence is diagnostic there — a live run heartbeats every 20s, a finished one re-offers every 10s. Route-driven turns re-attach too: without that, a re-applied trigger label dispatched a second run while the first held the answer.

An interrupted turn now reports a resumable pause, not an error.

2. Terminal runs that could never become terminal

markFailed — the only path that fails a run whose Job has vanished — hand-built its condition and appended it without LastTransitionTime, which the CRD requires:

AgentRun "10c84781-..." is invalid: status.conditions[0].lastTransitionTime: Required value

Every status update was rejected, so the run stayed Running, was re-reconciled every ~16 minutes, failed identically, and never became eligible for the retention sweep (which only reclaims terminal runs). Production held three AgentRuns and a ToolRun wedged this way for two days. Both Run controllers now use meta.SetStatusCondition, as every other controller in the package already did — it stamps the timestamp and replaces by type, which also closes the duplicate-Ready wedge (conditions is a map-list keyed by type). The wedged objects self-heal once this deploys.

3. The github image could never build

Two bugs in one layer. sha256sum -c <(grep ...) is a bashism that Debian's dash rejects outright — and underneath it, the tarball was saved as gh.tar.gz while sha256sum -c opens the name printed in the checksums file. This is what failed eff22d9b's release run and skipped its deploy.

This supersedes #166, which fixes only the bashism and would still fail on the filename mismatch — that PR was validated with dash -n but never built, so the second bug was invisible to it. Verified here by building the image and running gh --version (2.96.0) inside it.

4. A flaky HTTP test

Failed ~1 run in 10 with other side closed and zero bytes read — a request written to a dead socket, not a server fault. Node's fetch pools keep-alive sockets per origin; these tests listen on port 0 and close their servers, so a recycled ephemeral port hands the next test a corpse. Every request now sends connection: close (verified to defeat pooling: 3 requests, 3 sockets), applied to all four test files with that shape.

Verification

• Go controller suite via envtest: 18 specs pass, and the new orphaned-run specs fail without the fix — only a real API server rejects the update, so a fake client could never have caught this.
• All JS workspaces pass (orchestrator 483, agent-runtime 21, gateway 245).
• github image builds; gh runs inside it.
• Chart render confirms AGENT_REPLY_ACK_TIMEOUT_MS=90000 in e2e and absent by default.
• The new resilience.e2e.ts spec is written but not executed — it is guarded by requireMinikubeContext and this environment points at the live cluster.

Known gaps (recorded in the ADR, not hidden)

• The interrupted turn itself is still lost. The invocation record integration-gateway polls is an in-process Map; only the answer is recovered, on the next turn. Closing that needs durable invocation records.
• A mixed deployment holds pointlessly. An orchestrator too old to send reply_ack never acks, so agents from this commit hold every concluding message for the full timeout before exiting — bounded, correctness-neutral, and why the timeout is env-tunable.
• Agent-backed Tools (runTool) stay unresumable — no session slot to anchor them, matching the v1 cut already documented there.

🤖 Generated with Claude Code

https://claude.ai/code/session_01FUuaa2eY29JPocyiQZD9a8