Speed up the loader, make Redis persistence explicit, and reorganize loader docs

[gaurav]

Why

A full mode: load takes 0.5–1.5 days. Reading the loader together with the
translator-devops charts, the ceiling is the two single-threaded,
write-heavy Redis instances — eq_id_to_id_db and id_to_eqids_db
(150–220 GB) — that every compendium Job writes to at once. On top of that,
those instances ran --save 300 1000000, so during the write storm each hot DB
forked a multi-GB BGSAVE every ~5 min — pure copy-on-write waste, because the
backend DBs are write-once and persisted by a deliberate BGSAVE at the end.

Loader changes (low-risk, in-repo)

• Disable periodic RDB snapshots during load. load_all calls
  disable_periodic_save() → CONFIG SET save "" on each backend at load start.
  Best-effort (logs & continues if CONFIG SET is refused). Accepted trade-off: a
  crashed load leaves a DB empty (re-runnable), not partial.
• pipeline(transaction=False) everywhere — a bulk load isn't an atomic
  update, so the per-block MULTI/EXEC framing is pure overhead.
• Count source prefixes once per line (_accumulate_source_prefixes) instead
  of re-splitting every identifier once per Biolink ancestor type
  (was O(ancestors × identifiers)).

Deferred to v2.6.0 as issues (the riskier / bigger levers): MSET batching
(#387, held back because it's the one change that could silently drop data —
needs a correctness test first), redis-py 3.5.3→4/5 upgrade (#388), and
overlapping the four per-block flushes (#389).

Companion chart change — translator-devops #1044

Persistence is made explicit rather than periodic (no save <sec> <changes>
policy can snapshot only after a populate without also forking repeatedly
during it):

• redis-r3-external: save "" on all seven instances (no automatic snapshots),
  appendonly no kept. mode: load disables saves at runtime (this PR) and
  BGSAVEs manually at the end; mode: restore BGSAVEs at the end of run_pipe.sh.
• run_pipe.sh: the end-of-restore BGSAVE was fire-and-forget; it now waits
  for completion (LASTSAVE advances, rdb_last_bgsave_status:ok) and fails
  loudly on error/timeout, so a restored DB is guaranteed persisted before the
  Job exits. Addresses mode: restore BGSAVE is fire-and-forget; restored DB may not persist #390 (which will be closed when #1044 merges — GitHub can't auto-close it across repos).

Docs reorg

documentation/Development.md mixed loader internals, general dev setup, and the
frontend. Split into Loader.md (everything loader: the load process, the
write-once lifecycle, why persistence is explicit, the perf choices, a local
benchmark recipe, integration-test gotchas, the requests/docker landmine, the
v2.6.0 backlog) and CONTRIBUTOR.md (two-tools overview, running locally,
tests, a short Frontend section, the v2.5.0 backlog). References in CLAUDE.md
and the loader docstring updated.

Testing

• pytest -m "not integration" — loader unit tests pass, incl. a new
  test_accumulate_source_prefixes equivalence test.
• pytest -m integration — the loader integration test passes end-to-end through
  the new load_all (which exercises disable_periodic_save) against a real Redis.
• Verified against a real Redis that CONFIG SET save "" empties the save schedule.
• Local before/after benchmark on 2M lines of Protein.txt.02: ~11% from the
  client-side changes alone, output byte-identical (identical per-DB DBSIZE).
• Pre-existing failures in test_norm::test_empty / test_callback are unrelated
  (confirmed identical on the base commit).

🤖 Generated with Claude Code

[Copilot]

Pull request overview

This PR speeds up the NodeNorm loader’s Redis write path and makes Redis persistence behavior explicit during mode: load, while reorganizing developer/loader documentation to separate loader-specific operational guidance from general contributor setup.

Changes:

• Disable Redis periodic RDB snapshots during non-test loads (CONFIG SET save "") and remove pipeline MULTI/EXEC overhead via pipeline(transaction=False).
• Optimize per-line CURIE-prefix aggregation by counting prefixes once per line (_accumulate_source_prefixes) and add a unit test to pin equivalence.
• Reorganize docs: split prior Development.md into documentation/Loader.md and documentation/CONTRIBUTOR.md, and update references.

Reviewed changes

Copilot reviewed 7 out of 8 changed files in this pull request and generated 2 comments.

Show a summary per file

File	Description
tests/test_loader.py	Updates fake Redis pipeline signature and adds a unit test for _accumulate_source_prefixes.
node_normalizer/loader/loader.py	Adds disable_periodic_save(), uses pipeline(transaction=False) throughout, and optimizes prefix stats accumulation.
node_normalizer/loader/__init__.py	Exposes disable_periodic_save as part of the loader public API.
documentation/Loader.md	New loader-focused operational/performance doc, including explicit persistence rationale and testing notes.
documentation/Development.md	Removed in favor of the new split docs.
documentation/CONTRIBUTOR.md	New general contributor guide for running locally, tests, and frontend overview.
CLAUDE.md	Updates documentation pointers and loader behavior summary.
.gitignore	Ignores nodeNormalization-env/.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.