Presence/liveness tracks the guardian, not the harness: live harness reads as inactive; orphaned guardians hold the turn forever

[mostlydev]

Summary

Talking Stick tracks member activity and turn ownership against the wrong process. Two user-visible failures result:

1. A live harness can show as inactive in tt state, so peers effectively "can't see it" even though it is actively running tt commands.
2. A harness that exits without releasing leaves an orphaned guardian that renews the lease indefinitely — holding the turn forever and faking presence — so no peer can ever reclaim it without an operator takeover.

The shared root cause: presence/liveness is derived from a per-turn guardian process and from a pid that is never re-stamped, instead of from the durable harness and from actual harness activity. Guardian lease-renewal is conflated with harness presence.

This was diagnosed collaboratively by two agents (Claude + Codex) coordinating in the affected room. All findings below are verified against src/ at the current master.

---

Defect 1 — read/event paths register no presence

Read/observer RPCs (getRoomState, getEvents) call touchKnownMember (src/service.ts:1892), which early-returns unless a member row already exists:

private touchKnownMember(roomId, agentId, timestamp) {
  if (!agentId) return;
  if (!this.getMember(roomId, agentId)) return;   // <-- no row => no presence
  this.touchMember(roomId, agentId, timestamp);
}

Consequences:

• A live tt events --follow / tt events --wait / tt msg recv receiver that never ran tt join registers no presence at all — it is invisible in tt state.
• Even for a joined member, touchMember (src/service.ts:1832) only bumps last_seen_at and writes status='active'; it never re-stamps the live process metadata (pid / process_started_at / harness_*). So a stale pid is never repaired by ordinary commands.

Defect 2 — liveness inspects the guardian pid, never the harness

createDefaultProcessLivenessChecker (src/service.ts:2824) inspects only metadata.pid and ignores harness_pid entirely:

const inspection = inspector.inspect(metadata.pid);   // bare pid only
if (inspection === null || !inspection.startTime) return "gone";

But tt wait spawns a guardian via deriveHumanCliIdentity({ sessionKind: "human_guardian" }) (src/cli/guardian.ts:25), and that guardian's own pid is what ends up written into the member row. human_guardian has the highest sessionKindPriority (3, src/service.ts:2931), so shouldPreserveExactMemberProcessMetadata (src/service.ts:1802) keeps the guardian identity on the row over a lower-priority harness_cli update.

Consequences:

• When the guardian exits (e.g. after tt release), ps reports its pid dead → liveness "gone" → isMemberActive (src/service.ts:2407) returns false → the live harness shows inactive, even though harness_pid is alive and issuing commands.
• mapMember (src/service.ts:2696) recomputes status from liveness at projection time, so the status='active' column write from touchMember is moot.
• Only tt join / tt wait rewrite the member process metadata (via the big UPDATE at src/service.ts:1592), which is why a manual rejoin "repairs" visibility — and why nothing else does.

Defect 3 — the guardian never checks its own harness; heartbeat fakes presence

The guard loop (src/cli/guardian.ts:53-62) renews the lease on a fixed interval and never verifies its own harness_pid is still alive. It only exits on stale_lease / turn_mismatch / room_not_found:

const timer = setInterval(() => {
  try {
    runtime.commands.heartbeat(identity, heartbeatInput);   // renews lease, blindly
  } catch (error) {
    if (isProtocolError(error) && STALE_GUARD_ERRORS.has(error.code)) process.exit(0);
    process.exit(1);
  }
}, intervalMs);

A checkGuardianLiveness helper already exists (src/cli/guardian.ts:236) but is never applied to the guard's own harness.

Compounding this, heartbeat() calls touchMember (src/service.ts:620):

heartbeat(input) {
  ...
  this.touchMember(input.room_id, input.agent_id, timestamp);   // <-- guardian renewal bumps last_seen_at
  // ... UPDATE path_rooms SET lease_expires_at = ? ...
}

So an orphaned guardian's lease renewals keep last_seen_at perpetually fresh, meaning the abandoned session also passes hasRecentPresence and looks permanently active.

Net effect: a harness that exits without releasing orphans the turn forever. The detached guardian (reparented to init) keeps renewing the lease and faking presence; owner_timeout never fires because the lease never expires.

---

Reproduction (observed live)

Two co-located Codex sessions in the same room (/Users/wojtek/dev/ai/talking-stick):

• Abandoned session codex:e6fe0ade: harness pid 12800 (alive), guardian pid 49718 (alive, ppid 1 — reparented to init). Owns turn 17; guardian keeps renewing the lease.
• Live session codex:eb064c56: harness pid 46562 (alive). Wants to mutate but is blocked — it cannot release a turn it does not own (fragmented identity), and the abandoned owner's lease never expires.

Result: only an operator-authorized tt take can clear it. This is precisely the "abandoned-but-alive" case process-liveness cannot detect.

Minimal repro for Defect 1: run tt events --wait under a harness identity and observe the member's last_seen_at is unchanged before/after, while tt state's room inspector did refresh.

---

Expected contract

• Any non-guardian harness command refreshes visible presence (last_seen_at) and the member's current process metadata.
• Only wait / try additionally record turn-interest (last_wait_at).
• Only the guardian renews the lease, and guardian heartbeats must not count as harness presence (split the field — e.g. a separate guardian_seen_at/lease timestamp distinct from last_seen_at).
• Member liveness prefers harness_pid (+ harness_process_started_at), falling back to the bare pid only when no harness identity is present. This is safe because takeover is gated on lease_expires_at / claim_expires_at, not on member liveness, so owner_timeout continues to fire independently.

Proposed fix

A. Presence refresh on observer/read/event RPCs (Defect 1)

Introduce a presence upsert that takes caller process_metadata and writes last_seen_at + current harness metadata, but does not set last_wait_at and does not touch lease_expires_at. Apply it to observer/read/event RPCs.

Open design question (Claude vs Codex, undecided): should an observer command auto-create a member row for a harness identity that never ran join, or only refresh an existing row?

• Claude's position: do not silently join on one-shot reads (read-with-join side effect pollutes rooms with passers-by). Make presence a defined behavior of the sustained receive loop (events --follow / msg recv --follow) — that is the documented one-per-session presence primitive, so "watching" should register and refresh. One-shot state/notes refresh only if a row already exists.
• A newer same-harness session running an observer command should not trigger session_superseded/owner-clearing (that stays in joinPath, explicit participation) — a read must never reassign ownership.

B. Liveness prefers harness identity (Defect 2)

getMemberProcessLiveness should evaluate harness_pid/harness_process_started_at first and fall back to the bare pid only when no harness identity exists. Stop letting a dead guardian pid mark a live harness inactive.

C. Stale-guardian purge, two tiers (Defect 3)

• Tier-1 — harness dead (definitive, automatic, no timeout): the guard loop calls checkGuardianLiveness on its own harness_pid each interval; when the harness is gone, the guardian self-terminates and clears ownership. Service-side inspection should likewise treat an owner whose harness is provably dead as reclaimable regardless of a still-renewing guardian.
• Tier-2 — harness alive but idle: when the lease is still being renewed but the owner member's harness activity is older than ownerActivityTtlMs, surface takeover_available with a new reason (proposed owner_idle, not owner_timeout, since the process is alive).
  • Claude's refinement: gate Tier-2 on an actively-waiting peer — never disturb a long solo edit when nobody else wants the stick. The cost becomes asymmetric in the right direction.

Open design question — Tier-2 TTL (needs a decision)

• Codex: set ownerActivityTtlMs ≈ 2 × ownerLeaseTtlMs (~90 min) so a legitimate long edit with no tt calls is not reclaimed at exactly 45 min.
• Claude: with Tier-2 gated on an actively-waiting peer, ownerActivityTtlMs ≈ 1 × ownerLeaseTtlMs (~45 min) is safe — the peer-gating already protects long solo edits, so the 2× cushion is unnecessary.
• Also decide the takeover reason name: owner_idle (Claude) vs guardian_stale (Codex). Claude argues owner_idle because the guardian is healthy; it is the harness that is idle.

Suggested implementation order

1. Split guardian lease-renewal from harness presence (stop heartbeat() from bumping last_seen_at; add a separate guardian/lease timestamp).
2. Add the presence-upsert RPC + wire observer/read/event paths to it.
3. Make liveness prefer harness_pid.
4. Guard loop self-checks harness_pid → Tier-1 purge.
5. Tier-2 owner_idle takeover (peer-gated) + new takeover reason.
6. Tests: invisible receiver becomes visible; released-guardian member stays active while harness lives; orphaned-guardian Tier-1 auto-purge; Tier-2 peer-gated owner_idle; guardian heartbeat does not extend presence.

Affected files

• src/service.ts — touchKnownMember (1892), touchMember (1832), heartbeat (610-640), createDefaultProcessLivenessChecker (2824), getMemberProcessLiveness (2646), isMemberActive (2407), mapMember (2693), shouldPreserveExactMemberProcessMetadata (1802), sessionKindPriority (2931).
• src/cli/guardian.ts — guard loop (53-62), checkGuardianLiveness (236).
• src/identity.ts — deriveHumanCliIdentity / human_guardian metadata shape.

Note

The original "claude can't see codex" report that started this investigation was traced to a wrong-room/path mismatch (operator error), not a bug. The defects above are the genuine issues uncovered while investigating it.