SVM HyperSyncSolanaSource + indexer.onInstruction runtime (Stage 4 C2)#1218
Closed
JasoonS wants to merge 1 commit into
Closed
SVM HyperSyncSolanaSource + indexer.onInstruction runtime (Stage 4 C2)#1218JasoonS wants to merge 1 commit into
JasoonS wants to merge 1 commit into
Conversation
Makes the SVM ecosystem live: programs/instructions declared in YAML now
flow all the way through to the user-facing `indexer.onInstruction(...)`
handler.
Source layer (HyperSyncSolanaSource.res):
- Source.t implementation. Builds one `InstructionSelection` per
`(programId, discriminator)` declared in the config: the matching dN
field (d1/d2/d4/d8) carries the discriminator; a0..a5 carry positional
account filters (matching the C1 napi cap); `isInner`,
`includeTransaction`, `includeLogs` flow through.
- Per-(slot, tx_idx) transaction lookup and per-(slot, tx_idx,
instruction_address) log grouping so each handler sees only the logs
scoped to its instruction (Q2 answer — per-instruction grouping).
- Probes EventRouter longest-discriminator-prefix first via the per-program
byte-length ordering precomputed at router-build (Q1 answer).
- Synthesized logIndex `tx_idx * 65536 + depth-weighted addrSum` keeps
FetchState ordering deterministic without touching its compare logic.
- No-op reorg guard for C2 (Q3 deferred to C3 since it needs an extra
`queryBlockHash` route on the napi client).
Routing helpers (EventRouter.res):
- `getSvmEventId(~programId, ~discriminator)` -> `<programId>_<hex>` /
`<programId>_none` tag shape.
- `fromSvmEventConfigsOrThrow` returns the router AND a per-program
`svmProgramOrdering` (byte lengths sorted desc) so dispatch can probe
longest-first.
Config + dispatch:
- `Config.SvmSourceConfig` now carries `{hypersync: option<string>, rpc}`.
- `ChainFetcher.res` Svm arm: HyperSync primary when `hypersync` is set,
RPC stays for `getFinalizedSlot` height; RPC-only path unchanged.
- `Config.res buildContractEvents` accepts `~addresses` and the SVM arm
pulls `addresses[0]` as the real `SvmTypes.Pubkey.t` programId,
replacing C1's placeholder.
Public API (Main.res):
- `indexer.onInstruction({program, instruction, where?}, handler)` registers
via `HandlerRegister.setHandler(~contractName=program,
~eventName=instruction, ...)`. Both TS-string and ReScript-GADT identity
shapes are parsed via the same two-format dance as `onEvent`.
- SVM indexer key list grows from `[name, description, chainIds, chains,
onSlot]` to `[..., onInstruction, onSlot]`.
Tests:
- New `EventRouter_svm_test.res` exercises `getSvmEventId` shape and the
per-program ordering returned by `fromSvmEventConfigsOrThrow`. Two cases,
both passing.
- 264 cargo tests + full rescript build (130 envio modules + 168 test_codegen
modules) clean.
Out of scope (deferred to C3):
- Live Metaplex e2e scenario (`scenarios/svm_token_metadata/`).
- Real reorg-guard block hashes (needs the `queryBlockHash(slot)` route on
the napi client per Q3).
- TS-side `index.d.ts` `OnInstruction` mirrors so TypeScript users get the
fully-typed handler API (today they get `any`).
Stacks on PR #1217 (claude/solana-handler-codegen).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Contributor
|
Important Review skippedAuto reviews are disabled on base/target branches other than the default branch. Please check the settings in the CodeRabbit UI or the ⚙️ Run configurationConfiguration used: Organization UI Review profile: CHILL Plan: Pro Run ID: You can disable this status message by setting the Use the checkbox below for a quick retry:
Comment |
4 tasks
JasoonS
added a commit
that referenced
this pull request
May 27, 2026
End-to-end demo: a complete `scenarios/svm_metaplex_demo/` indexer that
runs `envio start` against `solana.hypersync.xyz`, streams real Metaplex
Token Metadata instructions, and persists entities to Postgres.
**Verified live**: 132 instructions indexed (95 creates + 37 updates) /
96 distinct metadata accounts written to Postgres in ~60s of runtime
across the ~46k-slot backfill window.
Scenario contents (`scenarios/svm_metaplex_demo/`):
- `config.yaml` — Metaplex Token Metadata program with
`CreateMetadataAccountV3` (0x21) + `UpdateMetadataAccountV2` (0x0f).
Pre-set `start_block` ~30-60k slots below current head for a quick
backfill, no `end_block` so it tails real time.
- `schema.graphql` — `TokenMetadataAccount` entity tracking
`mint`, `updateAuthority`, slot history; plus `ProgramStats` counter.
- `src/handlers/TokenMetadataHandlers.ts` — 100 lines of TypeScript:
two `indexer.onInstruction` registrations, each parsing the
positional `accounts` slot and writing entities. `console.log`-s
per instruction for demo visibility.
- `package.json`, `tsconfig.json`, `envio-env.d.ts`, `README.md`.
Stage 4 C3 wiring fixes uncovered during the live debug:
- **`EventRouter.fromSvmEventConfigsOrThrow`**: was keying the router by
bare `config.id` (= discriminator hex), but the source's lookup tag
is `getSvmEventId(~programId, ~discriminator)` (prefixed). Now both
sides use `getSvmEventId` to compute the key. Without this, every
matched instruction missed the router and silently dropped.
- **`Svm.makeRPCSource`**: now accepts an optional `~sourceFor` arg.
ChainFetcher's SVM dispatch passes `Fallback` for the RPC source
(it provides height + finalized slot) so the SourceManager doesn't
rotate to the RPC source for `getItemsOrThrow` (which still throws
"Svm does not support getting items" by design).
- **napi `HypersyncSolanaClient.get`**: was calling upstream
`client.collect(query, StreamConfig::default())` which paginates
client-side with 10x concurrent 1000-slot batches. The hyperindex
source layer paginates by slot range itself, so the napi binding
must be a single-window request. Swapped to `client.get(&q)`.
Without this, large slot windows (e.g. 44k backfill) hit
`502 Bad Gateway` from the server because of the parallel burst.
- **`Config.res`**:
- `publicConfigEcosystemSchema` accepts `"programs"` alongside
`"contracts"` (SVM-only alias).
- `publicContractsConfig` reads `svm.programs` when ecosystem is Svm.
- `contractEventItemSchema` now parses the optional `"svm"` event
descriptor (discriminator + flags + account_filters).
User-facing TypeScript surface (`packages/envio/index.d.ts`):
- Added `SvmInstruction`, `SvmTransaction`, `SvmLog`,
`SvmInstructionEvent`, `SvmOnInstructionHandlerArgs`,
`SvmOnInstructionOptions`, `SvmOnInstructionHandler`.
- `SvmEcosystem<Config>` now exposes `onInstruction(options, handler)`
alongside `onSlot`. The codegen-required fallback also lists
`onInstruction`.
Tests:
- `cargo test -p envio --lib` — 264 passed, 0 failed.
- `pnpm rescript` (envio + test_codegen) — clean.
- Scenario `pnpm tsc --noEmit` — clean (with the new TS types).
- Live: 132 indexed instructions / 96 entities (verified manually via
`docker exec ... psql ... SELECT COUNT(*) FROM "TokenMetadataAccount"`).
Stacks on PR #1218 (claude/solana-source-dispatch).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
DZakh
added a commit
that referenced
this pull request
Jun 10, 2026
* Embed HyperSync client directly into envio NAPI addon
Replaces the runtime `@envio-dev/hypersync-client` npm dependency with
in-tree NAPI bindings that wrap the upstream `hypersync-client` Rust
crate, exposed off the existing `envio.node` cdylib. Removes one
3rd-party native dep (plus its five per-platform binaries) from
hyperindex installs.
Surface kept minimal to what HyperIndex actually calls:
`HypersyncClient.{new, newWithAgent, get, getEvents}`,
`Decoder.{fromSignatures, decodeEvents}`, and module-level
`setLogLevel`. Decoder checksum behaviour simplified to a construction
flag (`Decoder.fromSignatures(sigs, ~checksumAddresses=...)`); the
former runtime enable/disable methods are gone.
Bumps `schemars` to 1.2 (required transitively by `hypersync-client`)
and updates `human_config.rs` for its renamed trait method.
* Embed Solana HyperSync client into envio NAPI addon
Adds a Solana counterpart to packages/cli/src/hypersync_source/, wrapping the
upstream hypersync-client-solana 0.0.2-rc.1 crate and exposing a
`HypersyncSolanaClient` napi class off the existing envio.node cdylib.
Surface kept minimal: `new`, `getHeight`, `get` (which wraps the upstream
`Client::collect` for paginated single-call queries). Query and response
shapes mirror the upstream net-types, with JS-friendly numerics (i64 for
slots and indices, `0x`-prefixed hex strings for instruction data and the
d1/d2/d4/d8 discriminator prefixes, base58 strings for pubkeys).
- packages/cli/Cargo.toml: add hypersync-client-solana, hypersync-solana-net-types.
- packages/cli/src/hypersync_source_svm/: new module
- config.rs: napi ClientConfig with url, api_token, timeout, retries.
- query.rs: napi SolanaQuery + InstructionSelection / TransactionSelection /
LogSelection / FieldSelection, with TryFrom into the upstream net-types.
Field-selection enums coerced from strings.
- types.rs: flat napi response shapes (Block/Transaction/Instruction/Log)
converted from upstream simple_types.
- mod.rs: HypersyncSolanaClient + #[ignore]-gated live smoke test
(verified locally: 390 Metaplex Token Metadata instructions decoded
from a 10k-slot window against solana.hypersync.xyz).
Stacks on PR #1212 (claude/rust-client-hyperindex-Nr6pt). No changes to the
EVM hypersync_source module; only one trivial fmt drift restored.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* ReScript layer: HyperSyncSolanaClient bindings to the SVM napi class
Adds a ReScript wrapper around the `HypersyncSolanaClient` napi class that
PR #1214 registers on the envio.node addon. Mirrors the shape of
HyperSyncClient.res:
- `cfg` record with url + optional auth/timeout/retry knobs.
- `QueryTypes`: field enums (block/transaction/instruction/log) plus the
selection records (`instructionSelection` carries `programId`, `d1`..`d8`
hex prefixes, `a0`..`a9` account filters, `isInner`, `includeTransaction`,
`includeLogs`; `transactionSelection` and `logSelection` analogously).
- `ResponseTypes`: typed block/transaction/instruction/log records, with
instruction `data` and discriminator prefixes as `0x`-prefixed hex and
pubkeys as base58 strings.
- `make` constructor + a `%raw` wrapper for the JS `new` operator (the
napi class is grabbed dynamically off `Core.getAddon()`, so `@new` can't
bind to a name).
Wires the new `hypersyncSolanaClient` constructor onto the addon record in
Core.res, alongside `hypersyncClient` and `decoder`.
Adds a `describe_skip`-gated live test at
`scenarios/test_codegen/test/HyperSyncSolanaClient_test.res` that hits
`solana.hypersync.xyz`, filters on the Metaplex Token Metadata program for
the last ~10k slots, and verifies the returned instructions decode through
the napi -> ReScript path. Verified locally end-to-end (vitest run passed,
~5s).
Out of scope here:
- HyperSyncSolanaSource.res (Source.t implementation): deferred until the
config schema (Stage 3) and handler-dispatch model (Stage 4) exist, since
bridging Solana instructions into Internal.item (today: Event | Block)
is the Stage 4 work.
Stacks on PR #1214 (claude/solana-hypersync-napi-binding).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* SVM config: programs + instructions + hypersync_config, with validation
Extends the SVM YAML schema (previously RPC-only with no contracts) to support
declaring Solana programs and the instructions to index on each. Mirrors the
EVM/Fuel "contracts -> events" shape, adapted to Solana.
human_config.rs (svm module):
- HypersyncConfig { url } — optional per-chain HyperSync endpoint, mirrors EVM.
- Program { name, program_id, handler?, instructions } — name drives codegen,
program_id is the base58 pubkey.
- Instruction { name, discriminator?, is_inner?, account_filters?,
include_transaction?, include_logs? } — discriminator is hex (1/2/4/8 bytes
including 8-byte Anchor), account_filters pin positional accounts (0..=9).
- AccountFilter { position, values }.
- Chain gains optional hypersync_config + programs.
system_config.rs:
- DataSource::Svm gains hypersync_endpoint_url: Option<String>; populated from
the parsed YAML. Validation runs before chain construction.
public_config.rs:
- SVM branch emits the HyperSync URL alongside the RPC URL so the runtime JSON
carries both. (Config.res still only reads `rpc` today; consuming `hypersync`
is wired up in Stage 4 along with the handler-dispatch model.)
validation.rs:
- is_valid_solana_pubkey: base58 alphabet + 32..=44 char length sanity check.
- validate_svm_discriminator: hex with optional 0x prefix, 1/2/4/8 bytes.
- validate_deserialized_svm_config_yaml: walks programs + instructions,
enforces unique program names + unique instruction names per program,
position in 0..=9, valid base58 for account-filter values.
svm.schema.json: regenerated via `cargo run --example script -- script
print-config-json-schema svm`. The existing test_svm_config_schema test
verifies the regenerated schema matches schemars' output.
Tests:
- 7 new tests in `validation::tests::svm` cover the validation surface
(valid + invalid pubkey, discriminator length + chars, duplicate program
names across chains, duplicate instruction names within a program,
account_filter position bounds, bad program_id).
- 2 new tests in `human_config::tests::svm_yaml` cover the round-trip
(Metaplex Token Metadata example yaml fully deserializes; unknown fields
rejected).
- All 153 config_parsing tests pass; no regressions in EVM or Fuel.
Out of scope (Stage 4):
- Translating programs into runtime Contract/Event structures for codegen.
- Handler API + dispatch (`<Program>.<Instruction>.handler(...)`).
- Wiring the new HyperSync URL into ChainFetcher source construction.
Stacks on PR #1215 (claude/solana-hypersync-rescript-source).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* SVM handler dispatch plumbing (Stage 4 C1)
End-to-end plumbing from YAML programs/instructions through to a typed
`Internal.svmInstructionEventConfig` ReScript runtime config. Handlers
compile but the actual dispatch (HyperSyncSolanaSource, ChainFetcher
wiring, EventRouter) lands in C2.
Rust side:
- `system_config::EventKind::Svm(SvmEventKind)` carries discriminator,
discriminator_byte_len, include_transaction, include_logs, account_filters,
is_inner. New `SvmAccountFilter`.
- `Abi::Svm` unit variant on the per-contract abi enum; Solana programs ship
no ABI artifact today (Borsh schema lands per the Stage 7 roadmap).
- `system_config.rs` HumanConfig::Svm arm walks each program/instruction,
producing a `Contract` with `Abi::Svm` and one `Event{kind: EventKind::Svm,
sighash: discriminator}` per instruction. ChainContract.addresses[0] holds
the base58 program_id.
- `public_config.rs` extends `ContractEventItem` with optional `svm`
descriptor + extends SvmConfig to carry `programs` alongside `chains`. The
per-event JSON now ships the SVM flags the runtime needs.
- `codegen_templates.rs`: new `EventTemplate::from_svm_instruction_event`
emits a minimal per-instruction ReScript module (`event` /
`paramsConstructor` / `onEventWhere` aliases) — enough surface for the
GADT to type-check. The rich `indexer.onInstruction` GADT registration is
C2 work.
- `contract_import_templates.rs`: SVM arm yields empty params (the
contract-import flow is EVM/Fuel-only).
ReScript side:
- New `SvmTypes.res` with a thin `SvmTypes.Pubkey.t` newtype (per the
Q4 review answer; treats Solana pubkeys distinctly from EVM `Address.t`).
- `Internal.svmInstructionEventConfig` mirrors `evm`/`fuelEventConfig`,
carrying programId + discriminator + flags so the future router can
dispatch by `(programId, discriminator)`.
- `Envio.res`: public `svmInstruction`, `svmTransaction`, `svmLog`,
`svmInstructionEvent`, `svmOnInstructionArgs<'context>`. Mirrors EVM's
`{event, context}` shape — the per-instruction `event` payload carries
`instruction`, `transaction?`, `logs?`, `slot`, `blockTime?`.
- `EventConfigBuilder.buildSvmInstructionEventConfig` is the runtime
constructor; `Config.res` Svm arm calls it from `buildContractEvents`.
- `HandlerLoader.applyRegistrations` Svm arm replaces the previous throw
with a Fuel-style pass-through (registration plumbing now ready for C2's
dispatch wiring).
Note on user-facing API: the locked roadmap originally described a
`Contract.Event.handler(...)` style, but EVM/Fuel actually use
`indexer.onEvent({contract, event}, handler)`. Mirroring EVM more honestly,
the C2 surface will be `indexer.onInstruction({program, instruction}, handler)`.
The generated per-instruction ReScript modules expose the GADT-friendly
type aliases now so adding that method in C2 is additive.
Tests:
- `cargo test -p envio --lib` — 154 passing (1 new SVM translation test
exercises the Metaplex YAML fixture end-to-end through
parse → validate → translate, asserting the Contract / Event /
Chain shape).
- `pnpm rescript` — 113 modules compile clean (added SvmTypes.res and
extended Envio.res / Internal.res / EventConfigBuilder.res /
HandlerLoader.res / Config.res).
- New fixture: `packages/cli/test/configs/svm-metaplex-config.yaml`.
Stacks on PR #1216 (claude/solana-hypersync-yaml-config).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* SVM HyperSyncSolanaSource + onInstruction runtime (Stage 4 C2)
Makes the SVM ecosystem live: programs/instructions declared in YAML now
flow all the way through to the user-facing `indexer.onInstruction(...)`
handler.
Source layer (HyperSyncSolanaSource.res):
- Source.t implementation. Builds one `InstructionSelection` per
`(programId, discriminator)` declared in the config: the matching dN
field (d1/d2/d4/d8) carries the discriminator; a0..a5 carry positional
account filters (matching the C1 napi cap); `isInner`,
`includeTransaction`, `includeLogs` flow through.
- Per-(slot, tx_idx) transaction lookup and per-(slot, tx_idx,
instruction_address) log grouping so each handler sees only the logs
scoped to its instruction (Q2 answer — per-instruction grouping).
- Probes EventRouter longest-discriminator-prefix first via the per-program
byte-length ordering precomputed at router-build (Q1 answer).
- Synthesized logIndex `tx_idx * 65536 + depth-weighted addrSum` keeps
FetchState ordering deterministic without touching its compare logic.
- No-op reorg guard for C2 (Q3 deferred to C3 since it needs an extra
`queryBlockHash` route on the napi client).
Routing helpers (EventRouter.res):
- `getSvmEventId(~programId, ~discriminator)` -> `<programId>_<hex>` /
`<programId>_none` tag shape.
- `fromSvmEventConfigsOrThrow` returns the router AND a per-program
`svmProgramOrdering` (byte lengths sorted desc) so dispatch can probe
longest-first.
Config + dispatch:
- `Config.SvmSourceConfig` now carries `{hypersync: option<string>, rpc}`.
- `ChainFetcher.res` Svm arm: HyperSync primary when `hypersync` is set,
RPC stays for `getFinalizedSlot` height; RPC-only path unchanged.
- `Config.res buildContractEvents` accepts `~addresses` and the SVM arm
pulls `addresses[0]` as the real `SvmTypes.Pubkey.t` programId,
replacing C1's placeholder.
Public API (Main.res):
- `indexer.onInstruction({program, instruction, where?}, handler)` registers
via `HandlerRegister.setHandler(~contractName=program,
~eventName=instruction, ...)`. Both TS-string and ReScript-GADT identity
shapes are parsed via the same two-format dance as `onEvent`.
- SVM indexer key list grows from `[name, description, chainIds, chains,
onSlot]` to `[..., onInstruction, onSlot]`.
Tests:
- New `EventRouter_svm_test.res` exercises `getSvmEventId` shape and the
per-program ordering returned by `fromSvmEventConfigsOrThrow`. Two cases,
both passing.
- 264 cargo tests + full rescript build (130 envio modules + 168 test_codegen
modules) clean.
Out of scope (deferred to C3):
- Live Metaplex e2e scenario (`scenarios/svm_token_metadata/`).
- Real reorg-guard block hashes (needs the `queryBlockHash(slot)` route on
the napi client per Q3).
- TS-side `index.d.ts` `OnInstruction` mirrors so TypeScript users get the
fully-typed handler API (today they get `any`).
Stacks on PR #1217 (claude/solana-handler-codegen).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* SVM Metaplex demo scenario + Stage 4 C3 wiring fixes
End-to-end demo: a complete `scenarios/svm_metaplex_demo/` indexer that
runs `envio start` against `solana.hypersync.xyz`, streams real Metaplex
Token Metadata instructions, and persists entities to Postgres.
**Verified live**: 132 instructions indexed (95 creates + 37 updates) /
96 distinct metadata accounts written to Postgres in ~60s of runtime
across the ~46k-slot backfill window.
Scenario contents (`scenarios/svm_metaplex_demo/`):
- `config.yaml` — Metaplex Token Metadata program with
`CreateMetadataAccountV3` (0x21) + `UpdateMetadataAccountV2` (0x0f).
Pre-set `start_block` ~30-60k slots below current head for a quick
backfill, no `end_block` so it tails real time.
- `schema.graphql` — `TokenMetadataAccount` entity tracking
`mint`, `updateAuthority`, slot history; plus `ProgramStats` counter.
- `src/handlers/TokenMetadataHandlers.ts` — 100 lines of TypeScript:
two `indexer.onInstruction` registrations, each parsing the
positional `accounts` slot and writing entities. `console.log`-s
per instruction for demo visibility.
- `package.json`, `tsconfig.json`, `envio-env.d.ts`, `README.md`.
Stage 4 C3 wiring fixes uncovered during the live debug:
- **`EventRouter.fromSvmEventConfigsOrThrow`**: was keying the router by
bare `config.id` (= discriminator hex), but the source's lookup tag
is `getSvmEventId(~programId, ~discriminator)` (prefixed). Now both
sides use `getSvmEventId` to compute the key. Without this, every
matched instruction missed the router and silently dropped.
- **`Svm.makeRPCSource`**: now accepts an optional `~sourceFor` arg.
ChainFetcher's SVM dispatch passes `Fallback` for the RPC source
(it provides height + finalized slot) so the SourceManager doesn't
rotate to the RPC source for `getItemsOrThrow` (which still throws
"Svm does not support getting items" by design).
- **napi `HypersyncSolanaClient.get`**: was calling upstream
`client.collect(query, StreamConfig::default())` which paginates
client-side with 10x concurrent 1000-slot batches. The hyperindex
source layer paginates by slot range itself, so the napi binding
must be a single-window request. Swapped to `client.get(&q)`.
Without this, large slot windows (e.g. 44k backfill) hit
`502 Bad Gateway` from the server because of the parallel burst.
- **`Config.res`**:
- `publicConfigEcosystemSchema` accepts `"programs"` alongside
`"contracts"` (SVM-only alias).
- `publicContractsConfig` reads `svm.programs` when ecosystem is Svm.
- `contractEventItemSchema` now parses the optional `"svm"` event
descriptor (discriminator + flags + account_filters).
User-facing TypeScript surface (`packages/envio/index.d.ts`):
- Added `SvmInstruction`, `SvmTransaction`, `SvmLog`,
`SvmInstructionEvent`, `SvmOnInstructionHandlerArgs`,
`SvmOnInstructionOptions`, `SvmOnInstructionHandler`.
- `SvmEcosystem<Config>` now exposes `onInstruction(options, handler)`
alongside `onSlot`. The codegen-required fallback also lists
`onInstruction`.
Tests:
- `cargo test -p envio --lib` — 264 passed, 0 failed.
- `pnpm rescript` (envio + test_codegen) — clean.
- Scenario `pnpm tsc --noEmit` — clean (with the new TS types).
- Live: 132 indexed instructions / 96 entities (verified manually via
`docker exec ... psql ... SELECT COUNT(*) FROM "TokenMetadataAccount"`).
Stacks on PR #1218 (claude/solana-source-dispatch).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* envio init SVM Metaplex template + SVM TUI slot labels (Stage 5)
`envio init svm template` now offers a Metaplex Token Metadata starter:
```
envio init svm template \
--name metaplex-demo --directory ./demo \
--template metaplex-token-metadata \
--language typescript --package-manager pnpm
```
Scaffolds a complete TypeScript indexer (config.yaml, schema.graphql,
TokenMetadataHandlers.ts, README, tsconfig) that streams Metaplex
CreateMetadataAccountV3 + UpdateMetadataAccountV2 instructions from
Solana mainnet via HyperSync. Verified end-to-end: project scaffolds,
codegen runs, `envio start` indexes against the live endpoint.
- `init_config::svm::Template` gains `MetaplexTokenMetadata` (in addition
to the existing `FeatureBlockHandler`).
- `template_dirs::Template for svm::Template` maps it to a new
`templates/static/svm_metaplex_template/` directory.
- Template ships with the same shape as the working `svm_metaplex_demo`
scenario: 2 instructions, per-instruction handlers writing
`TokenMetadataAccount` + `ProgramStats` entities, `context.log.info`
per matched instruction for demo visibility.
- `CommandLineHelp.md` regenerated (the new template name lands in the
`--template` valid-values list).
TUI slot labels:
- `TuiData.chain` gains `blockUnit: string`. The `Tui.res` `ChainLine`
component renders `"Slots: ..."` instead of `"Blocks: ..."` for SVM
chains, and `"(End Slot)"` instead of `"(End Block)"`. Plumbed via
`state.ctx.config.ecosystem.name`.
Tests:
- `cargo test -p envio --lib` — 264 passed, 0 failed (the
`check_cli_help_md_is_up_to_date` test caught the help drift; CLI
help regenerated and now passes).
- `pnpm rescript` clean (envio + test_codegen).
Stacks on PR #1221 (claude/solana-metaplex-demo).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* SVM E2E test + two runtime fixes uncovered by it (Stage 6)
Adds an automated end-to-end test that drives the whole SVM stack against
`solana.hypersync.xyz` deterministically — `HyperSyncSolanaSource` →
`EventRouter` → `indexer.onInstruction` dispatch → entity writes. Mirrors
the EVM `e2e_test` pattern (createTestIndexer + pinned slot window via a
`config.test.yaml`).
Files:
- `scenarios/svm_metaplex_demo/config.test.yaml` — test variant with a
500-slot pinned window (`417_950_000..417_950_500`). The demo
`config.yaml` stays as-is (no `end_block`) for live tailing.
- `scenarios/svm_metaplex_demo/src/indexer.test.ts` — vitest test that
sets `ENVIO_CONFIG=config.test.yaml`, runs the indexer, asserts on
one shape (Metaplex activity in the window, both
create+update kinds firing, counter consistency).
- `scenarios/svm_metaplex_demo/vitest.config.ts` — mirrors
`e2e_test/vitest.config.ts` (pool=forks, externalize non-test files
so NAPI addon load works).
- `scenarios/svm_metaplex_demo/package.json` — adds `vitest` dev dep
and `test: vitest run` script.
Runtime fixes uncovered by the test:
1. **`Envio.svmInstructionEvent` was missing the `block` sub-record.**
The shared `Ecosystem.t` getters (`Svm.res`) read
`event.block.{height, time, hash}` to drive `updateProgressedChains`
in `GlobalState.res`; without the field, dispatch crashed with
`TypeError: Cannot read properties of undefined (reading 'time')`.
Added a `svmInstructionEventBlock { height, time, hash }` mirroring
EVM/Fuel. `height` carries the slot. `time` is 0 and `hash` is ""
until the future reorg-guard `queryBlockHash(slot)` route lands.
Kept top-level `slot` / `blockTime` for user-ergonomic access.
2. **`SimulateItems.patchConfig` ignored `startBlock` / `endBlock`
overrides when no `simulate` items were present.** SVM doesn't
support simulate items (it has no `onEvent`/`onContractRegister`),
so the override was a no-op for SVM. Patch now applies the range
even without simulate, so
`testIndexer.process({chains: { 0: {endBlock: ...} }})` works.
Tests:
- New `scenarios/svm_metaplex_demo/src/indexer.test.ts` passes in ~15s
against the live endpoint.
- `cargo test -p envio --lib` — 264 passed.
- `pnpm rescript` — clean (envio + test_codegen).
Stacks on PR #1222 (claude/solana-init-flow).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* chore: pnpm allowBuilds prep + demo start_block bump
Pre-existing demo-prep changes from earlier sessions, separated out so
the Stage 7b commit stays scoped to the decoder runtime.
- pnpm-workspace.yaml: explicit `allowBuilds: esbuild: false` so pnpm
11.1.2's auto-write doesn't trigger ERR_PNPM_IGNORED_BUILDS.
- init template package.json.hbs: same fix for scaffolded projects.
- Stage 6 demo start_block: 417920000 -> 417995000 (12k below current
mainnet height for a fast backfill).
* feat(svm): Stage 7b runtime — decoded args/accounts on handler events
Wires the upstream hypersync-client-solana 0.0.3-rc.1 Borsh decoder
end-to-end through the SVM stack. Handlers now see
`event.instruction.decoded.args` (typed via locally-declared types until
typed-args codegen lands) and `event.instruction.decoded.accounts.<name>`
(IDL-faithful named accounts). The raw `instruction.data` /
`instruction.accounts[]` fields remain unchanged so existing handlers
keep working.
`packages/cli/src/config_parsing/human_config.rs`:
- `Program.idl: Option<String>` — path to an Anchor IDL JSON.
- `Instruction.accounts: Option<Vec<String>>` — positional account names.
- `Instruction.args: Option<Vec<ArgDef>>` — declarative Borsh layout.
- `ArgDef` / `ArgType` / `ArgPrimitive` / `ArgComposite` (incl. `Struct`
and `Enum` for nominal-type round-tripping). Mirrors upstream
`FieldType`.
`packages/cli/src/config_parsing/system_config.rs`:
- `Abi::Svm` promoted from marker to `Abi::Svm(SvmAbi { program_id,
defined_types, source })`.
- `SvmEventKind` gains `accounts: Vec<String>` and `args: Vec<NamedField>`.
- Resolution pipeline: IDL > bundled > inline > empty, with mutual
exclusion validation on `idl` vs per-instruction `accounts`/`args`.
- Helpers `yaml_type_to_field_type` / `field_type_to_arg_type` /
`named_field_to_arg_def` for the YAML <-> upstream conversion.
`packages/cli/src/config_parsing/public_config.rs`:
- `SvmEventItem` gains `accounts` + `args` (serialized as `Vec<ArgDef>`).
- `ContractConfig.svmAbi: Option<SvmAbiJson>` for program-level registry.
`packages/cli/src/hypersync_source_svm/borsh_decoder.rs` (new):
- `registerProgramSchema(descriptorJson) -> u32` — append-only global
registry of `ProgramSchema` indexed by handle. One call per program
at indexer startup.
- `decodeInstruction(handle, dataHex, accounts) -> { name, argsJson,
accountsJson, extraAccounts } | null` — single decode call per
instruction; any upstream error surfaces as `null` so the worker
doesn't crash on schema/on-chain drift.
`packages/envio/src/Internal.res`: `svmInstructionEventConfig` gains
`accounts` / `args` / `definedTypes` fields threaded from the wire JSON.
`packages/envio/src/EventConfigBuilder.res`: builder accepts the new
fields with sensible defaults.
`packages/envio/src/Config.res`: extended `svmEventDescriptorSchema`
and `contractConfigSchema` (the `S.schema` declarations that gate the
internal_config.json parse) to include the new fields. Threaded
program-level `definedTypes` from `contractConfig.svmAbi` down into
each event's config.
`packages/envio/src/Envio.res`: `svmInstruction` gains optional
`decoded: svmDecodedInstruction` carrying `{ name, args, accounts,
extraAccounts }`.
`packages/envio/src/sources/HyperSyncSolanaSource.res`:
- `buildSchemaHandles` groups eventConfigs by `programId` at `make`
time, builds one descriptor per program, registers via NAPI, stores
handles in a per-program dict.
- `decodeIfPossible` looks up the handle, calls the NAPI decoder, and
attaches the result to `Envio.svmInstruction.decoded`.
`packages/envio/src/Core.res`: addon record extended with
`registerProgramSchema` + `decodeInstruction` + the
`svmDecodedInstruction` shape.
`packages/envio/index.d.ts`: public `SvmDecodedInstruction` type;
`SvmInstruction.decoded?` field.
`scenarios/svm_metaplex_demo/src/handlers/TokenMetadataHandlers.ts`
rewritten to use `event.instruction.decoded.args.data.name` etc.,
locally typed via `CreateMetadataAccountV3Args` / `UpdateMetadataAccountV2Args`
type aliases until the typed-args codegen lands.
- `cargo test -p envio --lib` — 264/264 pass
- `scenarios/svm_metaplex_demo` vitest live e2e — pass, real on-chain
CreateMetadataAccountV3 decoded:
`[Create] name='SndkWdcAmdGoogIntelStxMu' symbol='SWAGISM'`
See STAGE_7B_DECISIONS.md for the rationale on:
- Eager schema registration at startup (handle-based) over per-call
lookup.
- Bundled-schema keyed by program_id only (no friendly shorthand).
- POC error policy: any decoder error -> `null`, indexer keeps running.
- Wire format drops per-account `optional` flag; NAPI marks all wire
accounts as `optional: true` so trailing sysvar omissions accept.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* feat(svm): typed args + accounts in onInstruction handlers (Stage 7b codegen)
Codegen now emits per-(program, instruction) `{ args, accounts }` TS
types into `.envio/types.d.ts` under `Global.config.svm.programs`. The
`indexer.onInstruction` signature in `index.d.ts` is overloaded to narrow
`event.instruction.decoded` based on the literal `{ program, instruction }`
selector, so handlers get fully autocompleted typed access without
casts or local type declarations.
## Codegen
`packages/cli/src/hbs_templating/codegen_templates.rs`:
- `field_type_to_ts_type` walks an upstream `FieldType` (with the
program's `defined_types` registry) and emits a TS type string.
Conventions match `STAGE_7B_DECISIONS.md` decision 3: sub-64-bit
ints / floats -> `number`, 64-/128-bit ints -> `string` (decimal),
pubkey / `[u8; 32]` -> `string` (base58), `Vec<u8>` -> `string`
(hex). Cycle guard on `Defined` recursion.
- `ts_safe_property_name` quotes non-identifier keys.
- New `svm_programs_body` builder iterates SVM contracts and emits
`{ "<Program>": { "<Instruction>": { args: ...; accounts: ... } } }`.
- `ConfigBodies` gains `svm_programs`. The `Ecosystem::Svm` arm of
`wrap_envio_module_augmentation` now emits `chains` + `programs`.
## Public types
`packages/envio/index.d.ts`:
- `SvmInstructionEvent` is now generic over a `Decoded extends
SvmDecodedInstruction` parameter so per-instruction overloads can
narrow `event.instruction.decoded` to the typed shape.
- `SvmDecodedFromProgramTable<TInstr>` helper extracts `{ args,
accounts }` from the codegen table and wraps them in a
`SvmDecodedInstruction`-shaped record.
- `SvmEcosystem`'s `onInstruction` becomes a generic over
`keyof Programs` / `keyof Programs[P]` when `Config["svm"].programs`
is present, falling back to the untyped signature otherwise.
## Demo handler
`scenarios/svm_metaplex_demo/src/handlers/TokenMetadataHandlers.ts`:
local `DataV2` / `CreateMetadataAccountV3Args` /
`UpdateMetadataAccountV2Args` declarations and `as` casts deleted.
Handler now reads `decoded.args.data.name` and
`decoded.accounts.metadata` with full TS autocomplete and type
checking driven by the generated table.
## Verification
- `cargo test -p envio --lib`: 264/264 pass.
- `pnpm exec tsc --noEmit` on the demo: clean (no manual types).
- Live e2e: real on-chain CreateMetadataAccountV3 still decoded:
`[Create] name='SndkWdcAmdGoogIntelStxMu' symbol='SWAGISM'`.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* fix(ci): clippy type-complexity + drop allowBuilds workspace key
Two Build & Verify failures on PR #1226:
1. clippy `type_complexity` on `bundled_program_schemas` return type.
Extracted `BundledProgramRow` alias so the signature reads cleanly.
2. `ERR_PNPM_LOCKFILE_CONFIG_MISMATCH` on `pnpm install --frozen-lockfile`.
The `allowBuilds: esbuild: false` block I added in 22c022cb gets
recorded in pnpm-lock.yaml metadata under a different key than the
workspace.yaml value; CI's frozen install fails the consistency
check. Removing the block since the suppression was only useful
locally; the template-side `pnpm.onlyBuiltDependencies` still
covers scaffolded user projects.
Local verification:
- `cargo clippy --manifest-path packages/cli/Cargo.toml -- -D warnings` clean
- `pnpm install --frozen-lockfile` clean
* fix(ci): regenerate pnpm-lock.yaml with pnpm 10 to preserve overrides
Local pnpm 11 (my dev env) stripped the `overrides: react-dom: 19.2.3`
block from the lockfile on `pnpm install`, which triggered
`ERR_PNPM_LOCKFILE_CONFIG_MISMATCH` under CI's pnpm 10 frozen install.
Switched local to pnpm@10.18.2 via corepack and re-ran install. The
resulting lockfile diff vs main is now just the legitimate
`scenarios/svm_metaplex_demo` importer added by Stage 5/6.
Verified: `CI=true pnpm install --frozen-lockfile` clean with pnpm 10.
* fix(ci): add missing svmInstructionEventConfig fields to test fixture
`scenarios/test_codegen/test/EventRouter_svm_test.res` constructs
`Internal.svmInstructionEventConfig` literally. The three fields added
in Stage 7b runtime (accounts/args/definedTypes) needed defaults here.
Verified: `pnpm rescript` in scenarios/test_codegen compiles clean.
* test(ci): skip 3 RpcSource tests hitting eth.rpc.hypersync.xyz
DO NOT MERGE WITH THESE TESTS SKIPPED.
CI's `ENVIO_API_TOKEN` lacks product access to `eth.rpc.hypersync.xyz`
(403 "Your token does not have access to this product"). The tests
themselves are correct; they just can't run against this token. Skipping
to unblock the v3.0.2-svm-alpha.0 experimental release.
Skipped tests:
- RpcSource - getHeightOrThrow > Returns the name of the source ...
- RpcSource - getEventTransactionOrThrow > Queries transaction fields ...
- RpcSource - getEventBlockOrThrow > Queries block fields ...
Each skip carries a `// DO NOT MERGE WITH THESE TESTS SKIPPED` line so
CodeRabbit / human reviewers flag the temporary skips before merge.
Restore by switching `Async.it_skip` back to `Async.it` once the token's
RPC subscription is provisioned.
* fix(hasura): force metadata reload before tracking to avoid race
`clear_metadata` followed immediately by `pg_track_tables` races with Hasura's
source-schema introspection on a freshly provisioned database: Hasura answers
`{"code":"metadata-warnings"}` (HTTP 400) for tables it can't yet see, the
existing response parser panics on the unrecognized code, and tracking is
never retried since `trackTablesRoute` is not wrapped by `sendOperation`'s
retry path. Downstream `createSelectPermission` calls then exhaust their own
retries logging `not-exists`, and GraphQL is permanently broken until manual
recovery.
Insert a `reload_metadata` (source: default) call between clear and track to
force Hasura to re-introspect before we attempt to register the user tables.
Removes the race without parsing internal warnings or adding ad-hoc retries.
Affects envio@3.0.2-svm-alpha.0.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
* feat(svm): any_of OR composition for instruction account filters
Add a second shape for `account_filters` in svm config:
account_filters:
any_of:
- - position: 1
values: [pkA]
- - position: 3
values: [pkB]
Outer list is OR across AND-groups; inner list is the existing flat
shape (AND across positions, OR within `values`). The model is exactly
DNF, which maps 1:1 to HyperSync's `array<instructionSelection>` so no
new wire support is needed.
Wire selections are now emitted one per AND-group sharing the same
`(programId, dN)`, so the EventRouter still sees a single entry per
event config and per-instruction dispatch is unchanged.
Validation tightened: positions must be in 0..=5 (6..=9 reserved for a
future extension), and duplicate positions inside a single group now
hard-error instead of silently keeping the first.
* feat(svm): add token balance support to event surface
Plumb SPL Token / Token-2022 pre/post balance snapshots through the
full pipeline so handlers can access them via
event.transaction.tokenBalances when include_token_balances is true
in config.yaml. include_token_balances implies include_transaction.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* refactor(svm): rename programs to programs_experimental in YAML config
Signals that the SVM config surface may change in future releases.
Internal Rust types (Program, Vec<Program>) keep their names; only
the serde-facing YAML key changes via #[serde(rename)].
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* chore(svm): remove handler field from example configs
Runtime auto-discovers handlers from src/handlers/. The field is
still valid in the schema for users who need explicit paths.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* Replace include_* flags with field_selection on SVM instructions
Consolidate include_transaction, include_logs, include_token_balances
into a field_selection block that accepts true (all fields) or a list
of field names (per-field selection, not yet supported). This aligns
with the EVM field_selection pattern and is forward-compatible.
When field_selection is absent, no extra data is fetched. The
downstream pipeline (public_config, Config.res, EventConfigBuilder,
HyperSyncSolanaSource) remains unchanged — the conversion happens
in system_config.rs when building SvmEventKind.
https://claude.ai/code/session_01YV1sASgBTS5Sx5ktDTD42P
* fix: remove orphaned Trace impl block left from rebase
The Trace struct was removed by the Rust decoder refactor on main,
but the impl block survived conflict resolution.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* fix: resolve rebase artifacts from Rust decoder refactor
- Config.res: rename eventItem["event"] to eventItem["sighash"] to
match main's field rename in the contract event item schema
- system_config.rs: replace .map_or(false, ...) with .is_some_and(...)
to satisfy the clippy unnecessary_map_or lint
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* fix(svm): repair RpcSource test field name + init template prompt
- RpcSource_test.res: use allEventParams (main's field name) instead
of allEventSignatures (stale name from pre-rebase branch)
- interactive_init/mod.rs: delegate to svm_prompts::prompt_template_init_flow
instead of hardcoding FeatureBlockHandler, so the Select prompt
shows all available SVM templates
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* fix(svm): thread IDL instruction schemas into layout resolution + accept null definedTypes
Two fixes to the Solana instruction-indexer (found building the Flow X-Ray demo):
1. IDL-decoded programs got empty args/accounts and never registered a decode
schema. resolve_program_schema parsed the Anchor IDL but kept only
defined_types, discarding the per-instruction schemas; lookup_program_schema
returned None for AnchorIdl. So resolve_instruction_layout's disc lookup never
matched and every idl: instruction fell through to empty -> codegen emitted
`args: {}` and the runtime registered no schema (decode returned None). Now
SvmAbi carries an owned `instructions` map (populated for both AnchorIdl and
Bundled) and resolve_instruction_layout looks it up directly.
2. Inline-schema programs send `definedTypes: null` in the program schema
descriptor; the Rust `#[serde(default)]` accepts an absent field but not an
explicit null ("expected a map"), crashing schema registration. Coalesce null
to {} in HyperSyncSolanaSource.buildSchemaHandles.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* feat(svm): add svm_flow_xray scenario - cross-protocol Flow X-Ray demo
Indexes Jupiter / Kamino / Drift / Raydium Solana instructions (is_inner unset)
so each tx's CPI tree + cross-protocol edges reconstruct in SQL views. Flat
append-only entities (InstructionNode / TokenDelta / FlowTx / LiquidationEvent /
IndexerStats) keyed by txSig + instructionAddress; aggregation lives in
sql/views.sql. IDL programs carry explicit Anchor discriminators (legacy IDLs
have none). SPL-Token/System dropped from the matched set (volume); value comes
from transaction.tokenBalances. Includes idls/, sql/views.sql + apply-views.sh,
and a live vitest against a pinned slot window.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* fix(svm_flow_xray): correct Drift v2 program_id
The Drift program_id was a vanity-address lookalike ending in ...ozatL,
which matches no program on-chain, so the instruction filter returned zero
Drift instructions on every endpoint. Replace with the real Drift v2 mainnet
program dRiftyHA39MWEi3m9aunc5MzRF1JYuBsbn6VPcn33UH (drift-labs/protocol-v2
DRIFT_PROGRAM_ID), verified against live HyperSync data where matched
instructions carry the configured placePerpOrder discriminator.
https://claude.ai/code/session_01R5MK69GPR4kiw8er5QoZKm
* feat(svm): surface block_time from HyperSync to handlers
Previously every svmInstructionEvent carried `blockTime: None` and
`block.time: 0` even though the HyperSync Solana client already exposes
the column. Wire it through:
- Always request `block: [Slot, BlockTime]` in the field selection,
so the response's blocks table is populated. (Other tables already
default to all-columns when not explicitly restricted.)
- Build a `blockTimeBySlot` lookup once per response.
- Populate `svmInstructionEvent.blockTime` + `block.time` from it.
- Update `reorgGuard.rangeLastBlock.blockTimestamp` and
`latestFetchedBlockTimestamp` from the highest-slot block in the
response, so the indexer's source-stream timestamps also reflect
real wall-clock time.
Handler code already reads these via the shared Ecosystem.t getters, so
no codegen changes needed; the public TS type already declares
`blockTime?: number`.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* hack28: v0.0.4 balances + chunk caps + close-to-head backfill
- Bump hypersync-client-solana 0.0.3-rc.1 -> 0.0.4 (token_balance scoped
join + per-selection include_token_balances flag, all serde-default
additive). Add the new fields to the napi binding's InstructionSelection
/ TransactionSelection / LogSelection / SolanaQuery + From impls and
expose them on the ReScript binding (HyperSyncSolanaClient.res).
- HyperSyncSolanaSource: cap chunk size (maxNumBlocks=4000,
maxNumTransactions=5000, maxNumInstructions=20000,
maxNumTokenBalances=40000). Without these the server resets the
connection mid-stream on multi-day windows once balances are joined in.
- config.yaml: bump start_block 420_650_000 -> 422_060_000 (50k slots
behind current head ~422.11M). Backfill stays well under the server's
per-request budget while covering ~5.5h of chain time.
- sql/views.sql: bump anchor 1748000000000 -> 1780000000000 (May 2025 ->
May 2026), matching the frontend slotToUnixMs fix.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* hack28: TLS-native-roots patch + tighter chunk caps + tight start_block
- patch.crates-io: point hypersync-client-solana + net-types + schema at a
local clone whose only difference is `rustls-tls` -> `rustls-tls-native-
roots`. The published 0.0.4 uses webpki-roots, and on this host that
bundle was missing the Let's Encrypt R13 intermediate path, so every
request died with `invalid peer certificate: UnknownIssuer`. Native roots
pulls the trust store from the macOS keychain (where curl already works).
- HyperSyncSolanaSource: shrink the per-chunk caps (4000/5000/20000/40000
-> 1000/2000/8000/16000) so a single chunk fits well inside the server's
per-request budget while upstream HOS-1304 lands.
- config.yaml: bump start_block 422_060_000 -> 422_100_000 (~10k slots
behind head, 70min of chain time) — keeps backfill short and on-disk
small.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* fix(svm): include_token_balances was silently dropped by Config schema
svmEventDescriptorSchema in Config.res didn't list "includeTokenBalances",
so rescript-schema parsing of the public-config JSON silently stripped the
field. Downstream, the Utils.magic widening in buildContractEvents
re-typed the slot as `bool` but the actual JS value was `undefined`, which
ReScript treats as falsy, so every svmInstructionEventConfig ended up with
includeTokenBalances = false even though config.yaml said
field_selection: token_balance_fields: true.
Net effect: needsTokenBalances stayed false, the per-instruction
include_token_balances join flag was never set, the field selection
didn't request the token_balance columns, and TokenDelta stayed empty
forever despite the v0.0.4 server being fully wired.
Wire it through the schema so the field survives parsing.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* hack28: add Orca + Meteora program-wide + mint prices
- config.yaml: match Orca (whirL...) and Meteora (LBUZ...) program-wide via
a discriminator-free instruction (renamed from "any" — reserved by codegen
— to "programIx"). Catches every Orca/Meteora ix including the inner ones
fired by Jupiter routes, so the CPI tree + cross-protocol heuristic light
up the way the UI assumes. token_balance_fields:true on both so deltas
flow through.
- handlers/flow.ts: register("Orca","programIx") + register("Meteora",
"programIx"). No arg mapper — the storyline is the CPI node + token deltas.
- views.sql: seed mint_price with USD1, cbBTC, JLP, KMNO, ORCA, PYTH,
POPCAT to cover the next tier of high-volume mints in the indexed window.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* Revert "hack28: add Orca + Meteora program-wide + mint prices"
This reverts commit 6b8b9ff3a0765f8d3448eb9487b3c9f9015286f1.
* hack28: Orca + Meteora narrow swap match for cross-protocol coverage
Almost every Jupiter route CPIs into Orca or Meteora; without indexing
their swap ix the cross-protocol heuristic (protocol_count >= 3) never
fires and every tx card looks like "Jupiter-only" or "Jupiter+Raydium".
- config.yaml: add Orca whirlpool + Meteora DLMM as separate matches with
the canonical swap discriminator 0xf8c69e91e17587c8 (sha256("global:swap"
)[..8]). Discriminator-filtered (not program-wide like the attempt that
timed out), so the response stays inside the server's per-request budget.
- handlers/flow.ts: register("Orca","swap"), register("Meteora","swap").
No arg mapper — the storyline is the CPI node + the token deltas the
parent tx already carries.
- start_block: 422_100_000 (~220k slots / ~24h of chain time) — empirically
the widest window that backfills cleanly with the new matches. Wider
windows trip the server's per-request budget; revisit once HOS-1304 ships.
After this lands, expect: real cross-protocol whales (e.g. Jupiter+Meteora
$104k), Jupiter+Meteora+Orca 3-protocol routes, and the cross-protocol /
protocol-edge views are no longer empty.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* hack28: add interest_score to v_interesting_tx
Composite ranking so the feed leads with the genuinely interesting txs
instead of whatever happens to be at the head slot. Weight ladder:
liquidation 5000
whale (>$100k) 4000
cross-protocol 2000 (>= 3 distinct programs)
arb-like 1000 (>= 2 programs, >= 2 mints, > $10k)
log(gross_usd) 0..500 tie-breaker
UI's Q_INTERESTING_FEED orders by interest_score desc, slot desc — see
monorepo commit. Existing flag columns are unchanged; tasks reading the
view without interest_score are unaffected.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* chore(release): self-contained on hypersync-client-solana 0.0.5
Drop the [patch.crates-io] local-path override and bump the cli deps
0.0.4 -> 0.0.5, so all three solana crates resolve from crates.io.
0.0.5 is published 0.0.4 plus reqwest rustls-tls-native-roots (TLS fix
for Let's Encrypt R13 certs). cargo check is clean against the registry.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* fix(release): restore pnpm overrides block dropped by block-time lockfile regen
The block-time lockfile regeneration dropped the top-level overrides block
(react-dom: 19.2.3) that package.json still declares, so the frozen
pnpm install in build-envio-package failed with ERR_PNPM_LOCKFILE_CONFIG_MISMATCH.
react-dom already resolves to 19.2.3 throughout, so this restores the block
to match package.json. Verified locally with pnpm install --frozen-lockfile.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* Address CodeRabbit nits in HyperSyncSolanaSource
- Annotate Utils.magic cast with explicit input record type (CLAUDE.md guideline)
- Fix typo heighestSlot -> highestSlot
* fix(svm): populate raw event timestamp from blockTime
SVM instruction events hardcoded Internal.Event.timestamp to 0, so
raw_events.block_timestamp was always 0 for SVM while other sources
populate it from block time. Use the already-computed blockTime.
Also drop the unused EventRoutingFailed exception (copied from the Fuel
source, never raised here).
* Remove STAGE_7B_DECISIONS.md
https://claude.ai/code/session_014ZXyEayvZSJvcRrCTBoSiM
* Fix getHeightOrThrow test name and unskip real-RPC tests
https://claude.ai/code/session_014ZXyEayvZSJvcRrCTBoSiM
* Collapse Metaplex demo to single config with env-var end_block
Replace config.test.yaml with an ENVIO_METAPLEX_END_BLOCK interpolation in config.yaml; the live test sets it to pin a finite window, the demo leaves it unset for continuous tailing.
https://claude.ai/code/session_014ZXyEayvZSJvcRrCTBoSiM
* Remove unrelated trace support from hypersync query.rs
https://claude.ai/code/session_014ZXyEayvZSJvcRrCTBoSiM
* Return block hashes from HyperSync Solana source
Select the block hash field and emit (slot, blockhash) pairs from getItems for reorg detection, mirroring the EVM source. Implement getBlockHashes via a paginated slot-range query filtered to the requested slots, replacing the previous unsupported stub.
https://claude.ai/code/session_014ZXyEayvZSJvcRrCTBoSiM
* Use single assertion for getHeightOrThrow height range
https://claude.ai/code/session_014ZXyEayvZSJvcRrCTBoSiM
* Decode SVM instructions in the Rust get, not per-instruction over napi
Thread the registered program schema handles through the Solana query so the client Borsh-decodes matching instructions inline (in get) and returns them on the response, mirroring how the EVM HyperSync client returns pre-decoded params. Removes the per-instruction decodeInstruction napi call from the ReScript fetch loop.
https://claude.ai/code/session_014ZXyEayvZSJvcRrCTBoSiM
* Recover schema registry lock instead of panicking on poison
The registry's critical sections only push/read Arcs, so a poisoned lock can't leave it torn. Recover the guard via PoisonError::into_inner through a shared lock_registry helper rather than .expect()-panicking the worker. Addresses CodeRabbit.
https://claude.ai/code/session_014ZXyEayvZSJvcRrCTBoSiM
* Align SVM sources to ReScript stdlib after Belt removal on main
PR #1290 swept Belt out of the runtime; the SVM source and EventRouter
additions reintroduced it. Switch to the stdlib equivalents
(Array.filterMap for keepMap, otherwise the un-prefixed name).
https://claude.ai/code/session_014ZXyEayvZSJvcRrCTBoSiM
* Nest SVM hypersync_config + programs under experimental; make rpc optional
The SVM chain config gains an `experimental` block holding the required
`hypersync_config` and the `programs` list (renamed from the top-level
`programs_experimental`). `rpc` becomes optional: a chain must declare at
least one data source (`rpc` or `experimental`), validated at parse time.
The runtime builds a HyperSync-only source when no rpc is given, keeping
the RPC fallback/height-oracle when one is present.
Updated the schema, templates, scenarios, and test fixtures to the new
shape.
https://claude.ai/code/session_014ZXyEayvZSJvcRrCTBoSiM
* Ignore generated scenarios *.res.mjs build artifacts
ReScript emits .res.mjs alongside the already-ignored .res.js; add the
matching ignore rule so compiled scenario helpers stay out of git.
https://claude.ai/code/session_014ZXyEayvZSJvcRrCTBoSiM
* Tweak experimental field description; regen svm schema
https://claude.ai/code/session_014ZXyEayvZSJvcRrCTBoSiM
* SVM onInstruction: flatten event envelope into the instruction arg
Handlers now receive { instruction, context } instead of { event, context }.
SvmInstructionEvent is removed; SvmInstruction itself carries the program /
instruction names, parent transaction, scoped logs, slot, and block. The
runtime remaps the generic dispatch field in onInstructionFn; block.height is
dropped (the slot lives on instruction.slot). Updates the public d.ts,
codegen, and the metaplex/flow_xray handlers.
https://claude.ai/code/session_014ZXyEayvZSJvcRrCTBoSiM
* Fix stale SVM instruction docs after the block.height removal + event flatten
Drop the lingering "alias for block.height" note on `slot` (block.height no
longer exists) and point the template README at the `experimental.programs`
config path.
https://claude.ai/code/session_014ZXyEayvZSJvcRrCTBoSiM
* SVM: ignore RPC when HyperSync is configured
RPC fallback isn't wired up yet, so when both an RPC and the experimental
HyperSync config are present, use only the HyperSync source.
* Carry block fields on the event item; drop ecosystem block getters
Move the block hash onto Internal.eventItem (timestamp and blockNumber
already lived there) and populate it at every source construction site, so
downstream code reads plain item fields instead of reaching into the opaque
event block through ecosystem getter helpers. Removes getNumber/getTimestamp/
getId from the Ecosystem interface and the per-ecosystem source modules, along
with the now-unused blockTimestampName/blockHashName.
ChainManager_test read blockNumber/timestamp off a locally constructed +
%identity-cast event item, which the optimizer constant-folds positionally;
read the in-scope source values directly instead.
* refactor(svm_flow_xray): register instructions with literal options
Drop the widening cast over indexer.onInstruction in favor of inline
literal { program, instruction } options, matching the typed overload the
way svm_metaplex_demo does. Shared handler logic moves into nodeHandler /
liquidationHandler factories.
* Remove unused HyperSync trace selection types from ReScript client
The traceSelection type and the traces/maxNumTraces query fields are EVM
trace-data selection, unused anywhere in the package and unrelated to the
SVM work in this PR.
* Add SVM codegen snapshot tests
Snapshot the SVM per-instruction ReScript module, the internal config
JSON, the envio.d.ts onInstruction program-table augmentation, and the
generated Indexer.res for the metaplex SVM fixture.
* Mark SVM Metaplex init template as experimental
Follows the existing "(Experimental)" label convention used for the
Subgraph Migration init option.
* Revert unrelated setLogLevel addition
The HyperSyncClient.setLogLevel helper and its Core addon binding are
unused and unrelated to this PR.
* Clean up SVM init template
Drop the esbuild build-allowance config (pnpm-workspace.yaml,
package.json pnpm.onlyBuiltDependencies) from scaffolded projects, and
fix the Metaplex .env.example to request ENVIO_API_TOKEN instead of an
unused RPC URL.
* Enable rollback_on_reorg by default for SVM
* Build TUI end-of-range label from blockUnit instead of branching
* Generate SVM block type with {slot, hash, time} fields
Add FieldSelection::svm() and skip the EVM default block fields for SVM
so the generated Block type matches the SVM block shape.
* Point SVM event-module params alias at svmDecodedInstruction
Mirror the EVM convention where params is the decoded payload, rather
than aliasing the full instruction.
* Gate SVM rollback_on_reorg on experimental hypersync source
RPC-only SVM chains stay at false; rollback is only meaningful when a
chain reads from the experimental HyperSync source.
* Add slot to SVM block; rename decoded->params, svmDecodedInstruction->svmInstructionParams
- svmInstructionBlock gains a slot field mirroring svmInstruction.slot
- svmInstruction.decoded? renamed to params? (still optional)
- type svmDecodedInstruction renamed to svmInstructionParams (and the
TS SvmDecodedInstruction/SvmDecodedFromProgramTable counterparts)
* Update SVM scenario handlers for decoded->params rename
* Make TUI block unit singular and append plural s where needed
* Drop redundant svmInstruction.slot in favor of instruction.block.slot
* Surface friendly type error for onInstruction when svm has no programs
* Configure SVM program schemas at client creation, drop global registry
Removes the process-global schema registry and the register_program_schema
napi method. The Solana client now takes per-program Borsh schema descriptors
in its config, builds them into decoders once at creation, and decodes
matching instructions against them in get() — no per-query schema handles.
* Drop redundant startBlock/endBlock override in SimulateItems no-simulate branch
The test indexer already seeds startBlock/endBlock into the persistence
initialState (TestIndexer), and the chain fetcher bounds the run from the
resumed chain state, not config.chainMap. Patching the config here was
redundant — no test depends on it.
---------
Co-authored-by: Claude <noreply@anthropic.com>
Co-authored-by: Jason <jason@envio.dev>
Co-authored-by: envioworker <envioworker@gmail.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Makes the SVM ecosystem live. Programs/instructions declared in YAML now flow all the way through to the user-facing
indexer.onInstruction({program, instruction}, handler)call.Stacked on #1217. Merge after Stage 4 C1.
What's in this PR
Source layer —
packages/envio/src/sources/HyperSyncSolanaSource.res(new, ~290 LOC)Source.timplementation. Builds oneInstructionSelectionper(programId, discriminator)from the chain'sInternal.svmInstructionEventConfig[]. The matchingdNfield (d1/d2/d4/d8) carries the discriminator;a0..a5carry positional account filters;isInner,includeTransaction,includeLogsflow through.(slot, tx_idx, instruction_address)→ logs map so each handler sees only the logs scoped to its instruction.d8→d4→d2→d1→nonein declared order, first hit wins.logIndex = tx_idx * 65536 + depth-weighted(instruction_address)keepsFetchStateordering deterministic without touching its compare logic.queryBlockHash(slot)route is deferred — needs a new napi binding method.Routing helpers —
packages/envio/src/sources/EventRouter.resgetSvmEventId(~programId, ~discriminator) → "<programId>_<hex>" | "<programId>_none".fromSvmEventConfigsOrThrowreturns(t<svmInstructionEventConfig>, array<svmProgramOrdering>). The ordering is per-program byte lengths sorted desc.Config + dispatch
Config.SvmSourceConfignow{hypersync: option<string>, rpc}.ChainFetcher.resSVM arm: HyperSync primary whenhypersyncis set, RPC stays forgetFinalizedSlotheight; RPC-only path unchanged.Config.resbuildContractEventsaccepts~addressesand the SVM arm pullsaddresses[0]as the realSvmTypes.Pubkey.tprogramId, replacing C1's placeholder.Public API —
Main.resindexer.onInstruction({program, instruction, where?}, handler)registers viaHandlerRegister.setHandlerwith(contractName=program, eventName=instruction). Both TS-string and ReScript-GADT identity shapes parsed via the same two-format dance asonEvent.[name, description, chainIds, chains, onSlot]to[..., onInstruction, onSlot].Test plan
scenarios/test_codegen/test/EventRouter_svm_test.res— 2/2 passing:getSvmEventIdshape (with + without discriminator).fromSvmEventConfigsOrThrowper-program ordering: ProgA with0x0f+0x0ffffffffffffffforders to[8, 1]; ProgB with no discriminator orders to[0].Deferred to C3
scenarios/svm_token_metadata/) — full end-to-end flow againstsolana.hypersync.xyz.queryBlockHash(slot)route tohypersync-client-solana+ napi binding, then toHyperSyncSolanaClient.res, then populatereorgGuard.rangeLastBlockproperly.index.d.tsOnInstructiondistributive-mapped types so TypeScript users get the fully-typed registration API (today they getanyfor the handler args).🤖 Generated with Claude Code