refactor: get indexer URLs from the indexing-payments subgraph#669
Merged
Conversation
Dipper queried the Graph network subgraph for a full topology of subgraphs, deployments and allocations, but only ever read indexer URLs out of it. A small poller against the indexing-payments subgraph (which records each indexer's URL on registration) replaces it.
CI's integration step selects tests named tests::it_ and fails when none match; this branch deleted the only 2 such tests along with the network subgraph client. New wiremock-backed tests drive the replacement fetch over HTTP: auth, pagination, invalid entries, and error responses.
The indexers query hardcoded first: 1000 while the pagination loop compared the page length against a separate PAGE_SIZE constant; editing one without the other would silently stop pagination after the first page. The query now reads the page size from the same constant.
The startup fetch retried without bound before the admin port opens, so a stalled subgraph left the pod hanging unready with no restart. Dipper now gives up after 6 attempts and exits, matching the bounded retry used for the RCA domain fetch.
The refresh loop keeps the previous snapshot when a fetch fails or returns zero indexers, but only the fetch function had tests. Drive the loop against a mock subgraph to cover the empty, failing, and successful refresh paths, and drop a needless borrow clippy flagged in this file.
The scheme check used starts_with("http"), which also accepted schemes like httpx://.
A subgraph endpoint answering with 0 indexers (a wrong but well-formed subgraph, say) let dipper boot with no URLs and then skip every selected indexer with warnings. Startup now retries on an empty snapshot and exits after 6 attempts, the same as a failing fetch.
Local networks legitimately boot before any indexer has registered, where the non-empty startup requirement would make dipper exit. With allow_empty_at_startup (default false) set, dipper boots with a warning and the refresh loop picks up the first registration.
MoonBoi9001
added
the
config-migration
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
This PR makes dipper look up indexer URLs in the
indexing-payments-subgraphinstead of thenetwork-subgraph, and deletes the Graph network subgraph client entirely.Dipper queried the network subgraph through the gateway to build a full topology of subgraphs, deployments, allocations and operators, but the only data ever read back was the indexer id to URL map. Every other consumer of the network subgraph was removed in earlier PRs. The payments subgraph records each indexer's service URL when it registers with the SubgraphService contract, so the new poller just reads that instead.
2 operator-visible changes:
networkconfig section now takes a single indexing payments subgraphsubgraph_endpointURL plus an optionalapi_key(thegateway_urlanddeployment_idkeys are gone)SubgraphService.Note the following consequences of this PR:
The indexer set changes The old code listed indexers from the network subgraph, which includes anyone registered on the legacy
ServiceRegistryor onSubgraphService. The new subgraph only contains indexers that emittedServiceProviderRegisteredonSubgraphService, and only from the deployed subgraph'sstartBlockonward.SubgraphServiceregistration anyway.Stale URLs never expire. While network subgraph nulls an indexer's URL when it unregisters from the legacy registry; the indexing-payments-subgraph Indexer entity is append-only because SubgraphService has no deregistration event. So the indexer set can only grow, and a departed indexer keeps its dead URL forever.
The operator filter was dropped. The prior solution excluded indexers that had no registered operators and no allocations. While the indexing payments subgraph has no operator data, so that filter is now gone.
Snapshot consistency and reorg detection are gone. The old paginated client pinned every page to a single block hash via _meta and detected reorgs by the "no block with that hash found" error. The new fetch pages with a plain id_gt cursor: meaning that pages can come from different blocks, and a reorged registration lingers until the next refresh cycle (60 seconds). The assumption is that a URL map doesn't need point-in-time consistency.