Agent-to-sidecar direct integration: heartbeats and bidirectional state

[kitplummer]

Summary

Flip the integration model from sidecar-polls-agent to agent-pushes-to-sidecar. The remote agent should send heartbeats directly to its co-located peat-sidecar, and query fleet state from it — enabling autonomous operation in DDIL.

Current State

Today the sidecar runs a watcher (src/watcher.rs) that polls the agent via Connect RPC:

• GET /status → writes to platforms/{agent-id}
• POST ListPackages → writes to deployments/{agent-id}:{pkg}
• POST ListPulledPackages → writes to packages/{agent-id}:{ref}

This is one-directional (agent → sidecar via polling). The agent has no awareness of the sidecar and cannot query fleet state from it.

Proposed Design

1. Agent → Sidecar (Heartbeats)

Instead of the sidecar polling the agent, the agent pushes heartbeats directly to the sidecar's Connect RPC API:

Remote Agent  --PutDocument("platforms", agent-id, heartbeat)--> peat-sidecar
Remote Agent  --PutDocument("deployments", agent-id:pkg, ...)--> peat-sidecar

This aligns with the existing AgentService.Connect() heartbeat pattern the agent already uses for the hub — the sidecar becomes a local heartbeat sink that replicates via CRDT.

Benefits:

• Agent controls heartbeat timing and content (no polling delay)
• Sidecar watcher becomes unnecessary for this path
• Agent can include richer state (workloads, labels) that the watcher can't see via current APIs

2. Sidecar → Agent (Fleet State Queries)

The agent queries the sidecar for fleet-wide state instead of reaching the hub:

Remote Agent  --GetPlatforms()--> peat-sidecar    (all fleet members' status)
Remote Agent  --GetCommands()--> peat-sidecar      (pending commands for this agent)
Remote Agent  --GetDocument("deployments", ...)--> peat-sidecar  (what other agents have)

This enables autonomous operation: When the hub is unreachable, the agent can still:

• See what packages other agents have (for peer-to-peer package transfer)
• Receive commands that were pushed into the mesh by the hub or peers
• Make deployment decisions based on fleet-wide state

3. Inverse State Flow (Hub → Agent via CRDT)

The hub writes desired state into the CRDT mesh (via its own peat-sidecar):

Hub API Server  --PutCommand(target=agent-1, deploy pkg-X)--> Hub peat-sidecar
                          ↓ CRDT sync
Edge peat-sidecar  ←←←←←←
                          ↓ agent queries
Remote Agent  --GetCommands()--> Edge peat-sidecar

The agent doesn't need to be "told" what to do — it queries its sidecar and acts on what it finds. This is the autonomous model described in the fleet management design.

Implementation

Phase 1: Agent SDK / Client Library

• Provide a Go client package (extend test/go/client.go) that the remote agent imports
• Thin wrapper around the Connect RPC API — agent calls PutPlatform(), GetCommands(), etc.
• Connect protocol means it's just HTTP + JSON — trivial to integrate

Phase 2: Agent Integration

• Remote agent imports the Go client
• On heartbeat interval, agent pushes status + workloads to sidecar
• Agent periodically queries sidecar for commands / fleet state
• Watcher becomes optional (backward compat for agents that haven't integrated yet)

Phase 3: Hub Integration

• Hub's peat-sidecar writes commands and desired state into the mesh
• Hub reads fleet state from its sidecar instead of (or in addition to) Postgres
• Graceful degradation: hub uses sidecar state when agents are DDIL-disconnected

References

• src/watcher.rs — current poll-based integration (to be superseded)
• proto/sidecar.proto — existing API already supports this (PutDocument, GetPlatforms, etc.)
• test/go/client.go — Go client that can be promoted to an importable package
• docs/DESIGN.md — fleet management architecture and DDIL problem statement
• uds-remote-agent#533 — future FleetService APIs