Update NodeNorm and NameRes to Babel 2025jul22#1044
Draft
gaurav wants to merge 62 commits into
Draft
Conversation
…tform/translator-devops into update-nodenorm-nameres-2025nov4
The redis-r3-external instances used `--save 300 1000000`, which forks a multi-GB BGSAVE every few minutes throughout a load — pure copy-on-write waste on the 150 GB+ databases. No `save <sec> <changes>` policy can snapshot only after a populate finishes without also forking repeatedly during it, so drive persistence explicitly instead: - redis-r3-external: set `save ""` (no automatic RDB snapshots) on all seven instances; keep `appendonly no`. mode:load disables saves at runtime and BGSAVEs manually at the end; mode:restore BGSAVEs at the end of run_pipe.sh. - run_pipe.sh: the end-of-restore BGSAVE was fire-and-forget, so a pod restart could reload a partial/empty dump.rdb. It now triggers BGSAVE and waits for it to complete (LASTSAVE advances, rdb_last_bgsave_status:ok) before the Job exits, failing loudly on error or timeout. Fixes NodeNorm issue #390. See NodeNorm documentation/Loader.md ("Why persistence is explicit, not periodic"). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
gaurav
added a commit
to NCATSTranslator/NodeNormalization
that referenced
this pull request
Jul 15, 2026
…loader docs (#391) ## Why A full `mode: load` takes 0.5–1.5 days. Reading the loader together with the `translator-devops` charts, the ceiling is the two **single-threaded, write-heavy** Redis instances — `eq_id_to_id_db` and `id_to_eqids_db` (150–220 GB) — that *every* compendium Job writes to at once. On top of that, those instances ran `--save 300 1000000`, so during the write storm each hot DB forked a multi-GB **BGSAVE every ~5 min** — pure copy-on-write waste, because the backend DBs are write-once and persisted by a deliberate BGSAVE at the end. ## Loader changes (low-risk, in-repo) - **Disable periodic RDB snapshots during load.** `load_all` calls `disable_periodic_save()` → `CONFIG SET save ""` on each backend at load start. Best-effort (logs & continues if `CONFIG SET` is refused). Accepted trade-off: a crashed load leaves a DB empty (re-runnable), not partial. - **`pipeline(transaction=False)` everywhere** — a bulk load isn't an atomic update, so the per-block MULTI/EXEC framing is pure overhead. - **Count source prefixes once per line** (`_accumulate_source_prefixes`) instead of re-splitting every identifier once per Biolink ancestor type (was `O(ancestors × identifiers)`). Deferred to **v2.6.0** as issues (the riskier / bigger levers): MSET batching (#387, held back because it's the one change that could *silently* drop data — needs a correctness test first), redis-py 3.5.3→4/5 upgrade (#388), and overlapping the four per-block flushes (#389). ## Companion chart change — translator-devops [#1044](helxplatform/translator-devops#1044) Persistence is made **explicit** rather than periodic (no `save <sec> <changes>` policy can snapshot only *after* a populate without also forking repeatedly *during* it): - `redis-r3-external`: `save ""` on all seven instances (no automatic snapshots), `appendonly no` kept. `mode: load` disables saves at runtime (this PR) and BGSAVEs manually at the end; `mode: restore` BGSAVEs at the end of `run_pipe.sh`. - `run_pipe.sh`: the end-of-restore BGSAVE was fire-and-forget; it now **waits** for completion (`LASTSAVE` advances, `rdb_last_bgsave_status:ok`) and fails loudly on error/timeout, so a restored DB is guaranteed persisted before the Job exits. **Addresses #390** (which will be closed when #1044 merges — GitHub can't auto-close it across repos). ## Docs reorg `documentation/Development.md` mixed loader internals, general dev setup, and the frontend. Split into **`Loader.md`** (everything loader: the load process, the write-once lifecycle, why persistence is explicit, the perf choices, a local benchmark recipe, integration-test gotchas, the requests/docker landmine, the v2.6.0 backlog) and **`CONTRIBUTOR.md`** (two-tools overview, running locally, tests, a short Frontend section, the v2.5.0 backlog). References in `CLAUDE.md` and the loader docstring updated. ## Testing - `pytest -m "not integration"` — loader unit tests pass, incl. a new `test_accumulate_source_prefixes` equivalence test. - `pytest -m integration` — the loader integration test passes end-to-end through the new `load_all` (which exercises `disable_periodic_save`) against a real Redis. - Verified against a real Redis that `CONFIG SET save ""` empties the save schedule. - Local before/after benchmark on 2M lines of `Protein.txt.02`: ~11% from the client-side changes alone, output byte-identical (identical per-DB `DBSIZE`). - Pre-existing failures in `test_norm::test_empty` / `test_callback` are unrelated (confirmed identical on the base commit). 🤖 Generated with [Claude Code](https://claude.com/claude-code)
Goes with NCATSTranslator/NameResolution#278 -- in that we can't make these changes until we switch to that process.
gaurav
changed the title
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
WIP