Add config/rule drift checks for all semantic-type id lists#728
Open
gaurav wants to merge 4 commits into
Open
Conversation
Raises WorkflowError at `snakemake --dry-run` if config.yaml protein_ids
lists a prefix with no rule producing intermediate/protein/ids/{prefix}.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
gaurav
changed the base branch from
master
to
add-pipeline-tests-for-shared-identifiers
April 22, 2026 06:14
Base automatically changed from
add-pipeline-tests-for-shared-identifiers
to
master
April 27, 2026 05:56
gaurav
marked this pull request as ready for review
June 4, 2026 08:25
Contributor
There was a problem hiding this comment.
Pull request overview
Adds a Snakemake-time validation to detect drift between config.yaml’s protein_ids list and the set of rules that actually generate intermediate_directory/protein/ids/{prefix}, so the workflow fails early (e.g., during --dry-run) when config expects an ID list that no rule produces.
Changes:
- Import
pathlib.Pathfor extracting output basenames. - Add a validation block that scans rule outputs under
intermediate_directory/protein/ids/and raises an error if anyconfig["protein_ids"]prefixes are uncovered.
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
|
|
||
| # import src.filter_compendia as filter | ||
| import src.snakefiles.util as util | ||
| from pathlib import Path |
Moves the inline config-drift validation from protein.snakefile into a shared util.find_missing_id_prefixes() so it can be reused across all semantic-type snakefiles without copy-pasting the set comprehension logic. Also removes the now-redundant `from pathlib import Path` import and trims the verbose inline comments that restated what the WorkflowError message already says. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Adds the find_missing_id_prefixes guard to anatomy, gene, chemical, diseasephenotype, process, taxon, and genefamily snakefiles so that any config.yaml *_ids entry without a matching rule fires a WorkflowError at DAG-construction time rather than failing silently mid-run. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
gaurav
changed the title
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Adds a
find_missing_id_prefixes()helper tosrc/snakefiles/util.pyanduses it in every semantic-type snakefile to verify that every prefix listed
in
config.yaml's*_idskeys has a corresponding Snakemake rule producingintermediate/<type>/ids/<prefix>. The check fires at DAG-construction time(including
--dry-run), so a config/rule mismatch is caught immediatelyrather than part-way through a multi-hour pipeline run.
Snakefiles covered
anatomy.snakefile—anatomy_idschemical.snakefile—chemical_idsdiseasephenotype.snakefile—disease_idsgene.snakefile—gene_idsgenefamily.snakefile—genefamily_idsprocess.snakefile—process_idsprotein.snakefile—protein_idstaxon.snakefile—taxon_idsChanges
src/snakefiles/util.py: newfind_missing_id_prefixes(workflow, ids_dir, config_list)— scansworkflow.rulesfor concrete outputs underids_dirand returns any prefixes fromconfig_listnot covered by any rule.WorkflowErrorif anything is missing.