Skip to content

fix(core): scheduler subscribers permanently stop on broadcast Lagged #535

Description

@emlautarom1-agent

Part of #402 — hardening follow-up to the app-wiring PR. Pairs with the retry-wrapper issue.

Scope

Scheduler duty/slot subscribers ride a bounded tokio::sync::broadcast channel (buffer 100) and permanently terminate on RecvError::Lagged (crates/core/src/scheduler.rs:128-137,161-169). A subscriber stuck for ~25 slots (e.g. hung beacon calls at 4 broadcasts/slot) silently kills all duty dispatch for that subscriber until the node restarts.

This is a Pluto-only failure mode: Charon dispatches per-duty goroutines and cannot lag. It is made reachable by the current serial subscriber processing (see the retry-wrapper issue, which restores async dispatch); fix both directions — a lagged subscriber should skip missed messages and continue with a warning (or subscribers should be made non-blocking per duty), never stop permanently.

Dependencies

None (the retry-wrapper issue reduces the likelihood but does not remove the failure mode).

Acceptance

  • A lagged subscriber logs a warning and continues receiving subsequent duties/slots.
  • Regression test: a slow subscriber that lags the channel resumes processing instead of terminating.

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't workingrustPull requests that update rust codetrack:duty-pipelineDev A — duty pipeline + consensus

    Type

    No type

    Fields

    No fields configured for issues without a type.

    Projects

    Status
    Todo

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions