Babel now distributes its compendia gzipped — compendia/Gene.txt.gz rather than compendia/Gene.txt — via a new rule compress_compendium (see NCATSTranslator/Babel#947). The loader cannot currently read those files, so a Babel release built from that change is not loadable until this is fixed.
What needs to change
node_normalizer/loader/loader.py opens compendia with a plain open() in two places:
- the validation read (
with open(in_file, "r") as compendium:)
- the main ingest read (
with open(compendium_filename, "r", encoding="utf-8") as compendium:)
Both need to pick gzip.open(path, "rt") when the filename ends in .gz. Babel's output is standard gzip (written by Python's gzip module), so nothing more exotic is needed — no custom framing, no multi-member handling beyond what gzip already does.
The conflation read (with open(f"{conflation_directory}/{conflation_file}", "r", encoding="utf-8")) is worth checking too, though conflation files are not currently compressed by Babel.
Context
This came out of PR #384, which made the loader substantially faster. That speedup is what made compressing the compendia attractive in the first place — the old SOP split the four largest compendia (Protein.txt, Publication.txt, SmallMolecule.txt, Gene.txt) with split -d -l 10000000 purely to work around slow loading. Worth re-checking whether splitting is still needed at all now.
Note that PR #384 itself did not touch file I/O — it changed loader structure and concurrency, so the open() calls above are unchanged from the previous implementation.
Interim workaround
Deployments running an older loader can gunzip the compendia before loading (Babel's docs/Deployment.md step 3.1 records this).
Babel now distributes its compendia gzipped —
compendia/Gene.txt.gzrather thancompendia/Gene.txt— via a newrule compress_compendium(see NCATSTranslator/Babel#947). The loader cannot currently read those files, so a Babel release built from that change is not loadable until this is fixed.What needs to change
node_normalizer/loader/loader.pyopens compendia with a plainopen()in two places:with open(in_file, "r") as compendium:)with open(compendium_filename, "r", encoding="utf-8") as compendium:)Both need to pick
gzip.open(path, "rt")when the filename ends in.gz. Babel's output is standard gzip (written by Python'sgzipmodule), so nothing more exotic is needed — no custom framing, no multi-member handling beyond whatgzipalready does.The conflation read (
with open(f"{conflation_directory}/{conflation_file}", "r", encoding="utf-8")) is worth checking too, though conflation files are not currently compressed by Babel.Context
This came out of PR #384, which made the loader substantially faster. That speedup is what made compressing the compendia attractive in the first place — the old SOP split the four largest compendia (
Protein.txt,Publication.txt,SmallMolecule.txt,Gene.txt) withsplit -d -l 10000000purely to work around slow loading. Worth re-checking whether splitting is still needed at all now.Note that PR #384 itself did not touch file I/O — it changed loader structure and concurrency, so the
open()calls above are unchanged from the previous implementation.Interim workaround
Deployments running an older loader can gunzip the compendia before loading (Babel's
docs/Deployment.mdstep 3.1 records this).