Conversation
…tput Every reason build_conflation() dropped a subject/object pair previously existed only as a logger.warning on STDERR, which Snakemake discards once the rule succeeds. That made regressions like #754 (the Carbidopa clique fragmenting between runs) impossible to diagnose after the fact. Add ConflationExclusionRecorder, which streams one row per dropped pair (source, reason, subject, object, subject_type, object_type, detail) to a gzipped TSV that is now a declared output of the drugchemical_conflation rule: reports/drugchemical/excluded_pairs.tsv.gz. Because it lands under reports/ it is published and retained every run, so "why was CHEBI:3395 dropped?" becomes a single zgrep against a past run. Instrument every drop site: the previously-silent RXNORM/UMLS else branch (the most likely place a bridge vanishes), all six PubChem-path filters, both manual-concord rejections, and the post-glom single-identifier collapse. A per-(source, reason) summary is logged and folded into metadata/DrugChemical.yaml. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Add offline unit tests covering the recorder round-trip (header, one row per dropped pair, per-(source, reason) counts) and the two manual-concord exclusion paths (CURIE not in any compendium, and self-pair after normalization). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Document how to investigate "these identifiers used to be conflated and now aren't" regressions, using #754 (Carbidopa) as the worked example: the confirmed 2024oct24 -> 2025sep1 regression, where conflation drops pairs, the new excluded_pairs.tsv.gz report, the deferred root-cause procedure (pending the PR #704 intermediate-concord Parquet export), and local-repro instructions. Cross-link from docs/Conflation.md. Also correct the stars.renci.org guidance in CLAUDE.md: use /var/babel_outputs/ (full identifiers; versions 2024oct24 and 2025sep1) for debugging, not /var/babel/ (externally-shareable subset only). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What this does
Makes DrugChemical conflation regressions diagnosable from retained build artifacts, motivated by
#754 (the Carbidopa clique fragmenting between the 2024oct24 and 2025sep1 runs).
Previously, every reason
build_conflation()dropped a subject/object pair existed only as alogger.warningon STDERR, which Snakemake discards once the rule succeeds. So a past run left notrace of why a cross-reference was dropped, and (because 2024oct24 published no
intermediate/tree) there was no way to diff inputs after the fact.
Changes
ConflationExclusionRecorder(src/createcompendia/drugchemical.py) streams one row perdropped pair —
source, reason, subject, object, subject_type, object_type, detail— to a gzippedTSV that is now a declared output of the
drugchemical_conflationrule:babel_outputs/reports/drugchemical/excluded_pairs.tsv.gz. Lands underreports/, so it ispublished and retained every run. Diagnosing "why was CHEBI:3395 dropped?" becomes one
zgrep.elsebranch (the mostlikely place a bridge vanishes), all six PubChem-path filters, both manual-concord rejections, and
the post-glom single-identifier collapse. A per-
(source, reason)summary is logged and foldedinto
metadata/DrugChemical.yamlunder anExclusionsblock.paths.
docs/debugging/Conflation.md(full diagnosis procedure, including the deferredroot-cause steps that depend on Export intermediate concords, identifiers, and metadata as Parquet files #704's intermediate Parquet export), cross-linked from
docs/Conflation.md. Corrected the stars.renci.org guidance inCLAUDE.mdto use/var/babel_outputs/(full identifiers) for debugging, not/var/babel/(shareable subset).Scope / not in this PR
This is the "do-now" instrumentation only. The actual #754 root-cause fix is deferred until the
intermediate-concord Parquet export (#704) lands, so the new exclusion report can be diffed against
the raw upstream edges; the procedure is documented in
docs/debugging/Conflation.md. Related TODOswere filed on #704. Also noted there: a separate duplicate-CURIE write bug observed in the 2025sep1
output, and instrumenting
build_rxnorm_relationships()'s concord-build-time drops.Testing
uv run pytest tests/createcompendia/test_drugchemical.py -q(9 passed). ruff, snakefmt, and rumdlall clean.
🤖 Generated with Claude Code