Skip to content

Hermes Health.ts degrades a healthy, quiet gateway: staleness is judged from the event-driven gateway_state.json instead of the 30s loop heartbeat #2044

Description

@lgertel

What the check does

LifeOS/install/LIFEOS/HERMES/Health.ts at main (ce046f2, release 7.40.4), lines 40 and 207–208:

const STATE_STALE_MS = 15 * 60 * 1000;

if (pidAlive && stateAgeMs !== null && stateAgeMs > STATE_STALE_MS) {
  problems.push(`gateway_state.json last written ${Math.round(stateAgeMs / 60000)}m ago`);
}

Any problem pushes status to degraded, so the menu bar row (modules/menubar.tshermesHealth()) and the /assistant card read gateway up · N/N channels · 1 issue for as long as the state file stays old.

What the gateway actually writes

gateway_state.json is event-driven. In NousResearch/hermes-agent (41447a6, 2026-08-25) it is written through gateway/status.py's n(...) writer, called from gateway/run.py on platform state changes and active-agent count changes — not on a timer. On a quiet channel it legitimately sits untouched for hours, and that is health, not staleness.

Liveness is proven elsewhere: gateway/shutdown_watchdog.py rewrites <HERMES_HOME>/state/gateway.heartbeat every 30 s:

47:DEFAULT_HEARTBEAT_INTERVAL_S = 30.0
58:_HEARTBEAT_RELATIVE = ("state", "gateway.heartbeat")

(the file holds {"pid", "updated_at", "monotonic", "start_time"}). Not to be confused with the LifeOS Heartbeat cron feature in HermesSidecar.md; this is the gateway main loop's own liveness stamp.

Observed

On a low-traffic install: pid alive, one channel connected, state/gateway.heartbeat written seconds before the check — and Health.ts reported 🟡 gateway up · 1/1 channels · 1 issue with gateway_state.json last written 881m ago, continuously for ~14 h. The gateway was fine the whole time; the instrument was reading the wrong file.

Suggested fix

When state/gateway.heartbeat exists, judge staleness from it (we used 5 min — ten missed beats — with the message loop heartbeat last written Nm ago — main loop hung), and fall back to the gateway_state.json age check only when the heartbeat file is absent (a gateway predating it). Add heartbeatUpdatedAt to HermesHealth; keep stateUpdatedAt as context. Down / flapping / fatal-platform detection unchanged.

Verified on a fork with three fixtures (stale heartbeat → degraded; dead pid → down; no heartbeat file → old fallback message) plus the live case above now reading 🟢 gateway up · 1/1 channels. Happy to open a PR if wanted.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions