fix: announce agreement events promptly and pause on outages#670
Merged
cmwhited merged 2 commits intoJul 14, 2026
Conversation
A recovery sweep running only every 60th poll was the sole emitter of the accepted, terminated and expired announcements, delaying each by 30 minutes while polling fast and up to 5 hours once the last live agreement was cancelled. Run those three every poll.
MoonBoi9001
added
the
agreement-events
MoonBoi9001
changed the title
Now that the accepted, terminated and expired announcements sweep every poll, a hung Kafka broker made each of the 3 sweeps block on its 30-second send, stalling chain polling every poll. After a failed send the sweeps now back off from 30 seconds up to 300 before retrying.
MoonBoi9001
force-pushed
the
mb9/emit-agreement-events-without-sweep-delay
branch
from
July 14, 2026 05:57
5c51ac2 to
6bacf8d
Compare
cmwhited
merged commit Jul 14, 2026
040845f
into
chris.whited/feat-producer/wire-into-dipper-svc
18 of 19 checks passed
cmwhited
added a commit
that referenced
this pull request
Jul 14, 2026
…emitter in callsites (#656) * feat(events producer): implement subgraph indexing agreements events emitter in callsites * feat(events producer): only emit n_indexer_unavailable once * fix: correct expiry timestamps and a duplicated shortfall event (#658) * fix(reassess): send shortfall event after the chain-time read The "not enough indexers" event was sent before the step that reads chain time to stamp deadlines. In local runs that read can fail and retry the whole job, resending the event each round; sending it after the read makes a retried round emit it once. * fix(expiration): report proposal time from the deadline clock The expiry event mixed clocks: "proposed at" came from wall time, "expired at" from chain time, so with local time controls on, expiry could look earlier than the proposal. Agreements now stamp proposal time in the deadline's clock; old rows fall back to created_at. * feat(events producer): PR review. hardening. atomic retries. error and app shutdown handling * fix: announce agreement events promptly and pause on outages (#670) * fix(listener): announce agreement events on every chain poll A recovery sweep running only every 60th poll was the sole emitter of the accepted, terminated and expired announcements, delaying each by 30 minutes while polling fast and up to 5 hours once the last live agreement was cancelled. Run those three every poll. * fix(listener): pause event sweeps briefly after a Kafka send failure Now that the accepted, terminated and expired announcements sweep every poll, a hung Kafka broker made each of the 3 sweeps block on its 30-second send, stalling chain polling every poll. After a failed send the sweeps now back off from 30 seconds up to 300 before retrying. --------- Co-authored-by: MoonBoi9001 <67825802+MoonBoi9001@users.noreply.github.com>
cmwhited
added a commit
that referenced
this pull request
Jul 14, 2026
…end events to kafka (#649) * feat(events producer): subgraph indexing agreement event emitter to send events to kafka * feat(events producer): cleanup. add simple tests to emitter * feat(events producer): implement subgraph indexing agreements events emitter in callsites (#656) * feat(events producer): implement subgraph indexing agreements events emitter in callsites * feat(events producer): only emit n_indexer_unavailable once * fix: correct expiry timestamps and a duplicated shortfall event (#658) * fix(reassess): send shortfall event after the chain-time read The "not enough indexers" event was sent before the step that reads chain time to stamp deadlines. In local runs that read can fail and retry the whole job, resending the event each round; sending it after the read makes a retried round emit it once. * fix(expiration): report proposal time from the deadline clock The expiry event mixed clocks: "proposed at" came from wall time, "expired at" from chain time, so with local time controls on, expiry could look earlier than the proposal. Agreements now stamp proposal time in the deadline's clock; old rows fall back to created_at. * feat(events producer): PR review. hardening. atomic retries. error and app shutdown handling * fix: announce agreement events promptly and pause on outages (#670) * fix(listener): announce agreement events on every chain poll A recovery sweep running only every 60th poll was the sole emitter of the accepted, terminated and expired announcements, delaying each by 30 minutes while polling fast and up to 5 hours once the last live agreement was cancelled. Run those three every poll. * fix(listener): pause event sweeps briefly after a Kafka send failure Now that the accepted, terminated and expired announcements sweep every poll, a hung Kafka broker made each of the 3 sweeps block on its 30-second send, stalling chain polling every poll. After a failed send the sweeps now back off from 30 seconds up to 300 before retrying. --------- Co-authored-by: MoonBoi9001 <67825802+MoonBoi9001@users.noreply.github.com> --------- Co-authored-by: MoonBoi9001 <67825802+MoonBoi9001@users.noreply.github.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The chain listener polls the subgraph for agreement state and announces each accepted, terminated or expired agreement on a Kafka topic that downstream consumers watch. This PR makes those announcements go out on every poll instead of every 60th, and adds a short cooldown so a Kafka outage cannot stall the poller.
Until now the 3 announcements were emitted only by a recovery sweep that runs every 60th poll. At the active 30-second poll rate an announcement could trail its state change by up to 30 minutes; worse, cancelling the last live agreement drops the listener to its 5-minute idle poll rate, parking that final termination announcement for up to 5 hours. Running the sweeps every poll closes that gap: each per-event query is backed by a partial index, so an empty sweep is cheap.
Sweeping every poll makes a broker outage more expensive, though. Each sweep sends synchronously with a 30-second timeout, so a hung broker would block all 3 sweeps for up to 90 seconds on every poll and fill the logs with warnings. After a failed send the listener now pauses the event sweeps, backing off from one poll interval (30 seconds) up to 5 minutes and retrying, then resumes once a send succeeds. The cancellation recovery sweeps, which do not send Kafka events, and the healthy path where the broker is up are both unchanged.
A separate review finding about diagnostic events being dropped when the in-memory send queue fills is already handled on this base branch: the queue capacity is configurable (default 1,024) and every drop is counted and logged. No change was needed for it here.