Skip to content

feat/store: improve node store startup time - 2nd round#1802

Draft
drahnr wants to merge 3 commits intonextfrom
bernhard-improve-node-startup
Draft

feat/store: improve node store startup time - 2nd round#1802
drahnr wants to merge 3 commits intonextfrom
bernhard-improve-node-startup

Conversation

@drahnr
Copy link
Contributor

@drahnr drahnr commented Mar 17, 2026

Ref #1697

Adds a bench with progress bars and prints closing spans, which shows "the numbers" for State::load pieces.

@drahnr
Copy link
Contributor Author

drahnr commented Mar 25, 2026

The bisection of last week and performance pointed to reconstructing the LargeSmtForest from the SQL as the by far leading cause, specifically fn add_lineage which effectively linearizes account additions to the forest (bad).

Run the benchmark RUST_LOG=info cargo bench --bench state_load -- --nocapture to see span based logs:

  INFO state_load: message=Reusing cached data directory, num_accounts=100000
Benchmarking state_load/100000_accounts: Warming up for 3.0000 sINFO load: load: miden-store: message=Connected to the database, sqlite=/media/humpback/projects/miden/miden-node-bernhard-improve-node-startup/crates/store/../../target/bench-data/100000/miden-store.sqlite3
INFO miden_node_store::db::migrations: message=Applying migrations, target="miden-store", migrations=0
INFO verify_schema: miden-store: message=Database schema verification passed, objects=37
INFO verify_schema: miden-store: message="close", time.busy=1.31ms, time.idle=1.79µs
INFO load: load: miden-store: message="close", time.busy=196µs, time.idle=2.49ms
INFO load: load_mmr: miden-store: message="close", time.busy=17.6µs, time.idle=46.0µs
INFO load: load_account_tree: miden-store: message="close", time.busy=868ms, time.idle=2.30µs
INFO load: load_nullifier_tree: miden-store: message=RocksDB nullifier tree storage is empty, populating from SQLite
INFO load: load_nullifier_tree: miden-store: message="close", time.busy=406ms, time.idle=1.89ms
INFO load: verify_tree_consistency: miden-store: message="close", time.busy=11.4µs, time.idle=202µs
paging       [=============================>                    ] 59000/100000 (73s remaining)
fetching     [============================>                     ] 56000/100000 (85s remaining)
inserting    [===========================>                      ] 54000/100000 (84s remaining)  

A bunch of optimizations are already done:

  • Batch fetching from DB and processing - keep icaches warm while utilizing many more cores
  • Limited parallelism
  • Staged work
  • Backpressure

What is still required to land this:

0xMiden/crypto#910 which allows to batch apply lineages and avoids a lot of intermediate hash re-calc on addition.

If things are still looking somewhat bleak, having a begin() -> Operator; Operator::add_lineage; Operator::drop/end()-style approach, avoiding all intermediate node operations would be my next steps to take

@Mirko-von-Leipzig since 0xMiden/crypto#910 is only added in next for now, I'll not be able to close this one

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant