dataset: synthetic from PANTHER by tristan-f-r · Pull Request #25 · Reed-CompBio/spras-benchmarking

tristan-f-r · 2025-07-01T18:44:17Z

This does not add anything to config/*.yaml.

Co-Authored-By: Neha Talluri 78840540+ntalluri@users.noreply.github.com
Co-Authored-By: Oliver Faulkner Anderson 112665860+oliverfanderson@users.noreply.github.com
Co-Authored-By: Altaf Barelvi altafayyubibarelvi@gmail.com

Co-Authored-By: Neha Talluri <78840540+ntalluri@users.noreply.github.com> Co-Authored-By: Oliver Faulkner Anderson <112665860+oliverfanderson@users.noreply.github.com> Co-Authored-By: Altaf Barelvi <altafayyubibarelvi@gmail.com>

ntalluri · 2025-07-01T20:18:37Z

datasets/synthetic-data/Snakefile

@@ -0,0 +1,100 @@
+pathways = ["Apoptosis_signaling", "B_cell_activation",


Each of the files have this variable. I think we should have it only in the snakefile and send this list to each of the files that use this pathway list

datasets/synthetic-data/Snakefile

ntalluri · 2025-07-03T18:16:43Z

A question I’d appreciate feedback on: Currently, we generate separate source, target, and prize files for each pathway, but we combine all pathways into each thresholded interactome. Should we also create a combined list of sources, targets, and prizes? Should we also combine the gold standard as well? Or would it be better to keep separate interactomes for each individual pathway (keep it the way it is)?

tristan-f-r · 2025-07-28T18:11:41Z

We should have separate gold standards.

ntalluri · 2025-07-28T18:33:18Z

When this is reviewed (or before) we should do tests to see how connected the networks are after thresholding, adding back the pathway data, and removing proteins that don't have uniprot ids.

ntalluri · 2025-07-28T18:52:38Z

Also there is a chance we can use more panther pathways, we should look to see what else we can use from pathway commons.

ntalluri · 2025-07-31T16:35:50Z

@oliverfanderson @ctrlaltaf For the gold standard nodes (and potentially the edges), should we exclude source, target, and prize nodes when defining it? Currently, it looks like we’re including these nodes in the gold standard for each pathway. These nodes overlap with the gold standard, but that overlap should happen naturally, not by construction/being predefined. I’m concerned this could inflate our precision and recall metrics, because of a form of data leakage.

ntalluri · 2025-08-11T21:25:55Z

@oliverfanderson @ctrlaltaf For the gold standard nodes (and potentially the edges), should we exclude source, target, and prize nodes when defining it? Currently, it looks like we’re including these nodes in the gold standard for each pathway. These nodes overlap with the gold standard, but that overlap should happen naturally, not by construction/being predefined. I’m concerned this could inflate our precision and recall metrics, because of a form of data leakage.

Plan to keep all of them in the gold standard. But update the evaluation code to deal with the sources/targets/prizes being in the gold standard and shown as a different baseline where those are all set as frequency 1.0.

ntalluri · 2025-08-13T19:33:24Z

Should we also consider how sparse an interactome becomes after applying a threshold to the STRING interactome? When we filter by size, we implicitly accounting for the decrease in graph density as well. Would it make more sense to treat size and density as separate variables when evaluating performance? However, does testing for density even matter in this context; are there any interactomes that aren’t already highly connected?

I’m thinking we should first threshold the interactomes, then select only those that are highly connected (e.g., density ≥ 0.85). From that subset, we could choose a few to represent different size scales.

ntalluri · 2025-11-06T20:13:46Z

I will be updating how we create interactomes for the Panther pathways dataset.

Current:
Our current thresholded interactomes are built by applying a hard cutoff on experimental scores (keeping edges with score ≥ x).
While this approach retains high-score edges and removes low-score ones, it distorts the original score distribution, which could be problematic for algorithms that rely on edge scores during optimization.

New:
Instead for the interacomes, we should approach it in a downsampling approach: build smaller interactomes that preserve the original score distribution of the original network while also reducing the total edge count.

For example, in the STRING interaction networks, when using only physical interactions and experimental edge scores, we could aim to keep 25% of all edges.
To achieve this, we could:

Sample 25% (or remove 75%) of edges uniformly at random, ignoring edge scores
Or use stratified sampling by edge score bins to preserve the distribution of scores

Now we will be construct new interactomes by removing X% of edges and then adding all edges from all chosen PANTHER pathways. We will only keep downsampled interacomes that satisfy specified properties for a given set of sources and targets.

Proposed brute-force method for Panther pathways interactomes:

edge removal

Randomly remove X edges from the full STRING interactome

Option A: Remove edges uniformly at random (ignoring scores)
Option B: Stratify edges by score bins and sample within each bin to maintain the overall score distribution
- We can stratify the edges into bins based on their score ranges (example: [0–300], [300–600], [600–900] ...).
- Then, when we randomly remove X edges, we can remove them proportionally from each bin.
- Example: suppose 20% of the original edges fall in the 0–300 bin, 50% in the 300–600 bin, and 30% in the 600–900 bin. If we're removing 1000 edges total, we'd randomly pick ~200 from the first bin, ~500 from the second, and ~300 from the third.

pathway integration
Add all edges from the selected Panther pathways to the new downsampled interactome
Property checks

Verify that the new network maintains the following properties:

All-in-one: all sources and targets lie in the same connected component
- V_{ST} = {all sources} U {all targets}, and after edge removal, all vertices in V_{ST} should belong to the same connected component of the graph
Reachability: every target is reachable from at least one source (can be checked via apsp or bfs)
Might want to soften the criteria to be X% of the sources and targets remain in a single component,
and (potentially) X% of the targets are reachable from at least one source.

Restart is necessary

If the properties above are not satisfied, repeat the process with a different random sample.

ntalluri · 2025-11-06T20:13:52Z

For this dataset, we are planning on using it for all of the evaluations. I was deciding if we need to use all of the pathways, and I don't think we need to. I decided on a couple that we can use:

Balanced
Interleukin_signaling - 86 GS Nodes, 811 GS Edges, 18 S / 16 T (ratios 0.209 / 0.186)
Apoptosis_signaling - 108 GS Nodes, 286 GS Edges, 6 S / 17 T (0.056 / 0.157)

Skewed
Cadherin_signaling - 150 GS Nodes, 2650 GS Edges, 17 S / 3 T (0.113 / 0.02)
PDGF_signaling - 125 GS Nodes, 764 GS Edges, 2 S / 28 T (0.016 / 0.224)
Toll_signaling - 44 GS Nodes, 84 GS Edges, 8 S / 3 T (0.182 / 0.068)

Tiny
Hedgehog_signaling - 19 GS Nodes, 61 GS Edges, 2 S / 2 T (0.105 / 0.105)

When making the interactomes, I want to add all of these pathways on the thresholded interactomes and uphold the properties above.

I need to double check if I used any of these will break the rules for pilot data/runs; but since we are making a new dataset that wasn't used for my thesis, I think we will be okay.

tristan-f-r · 2026-01-24T05:10:53Z

Made minor changes to fix the interactome fetching - these shouldn't cause any conflicts, nor were the changes I wanted to make as mentioned in Slack. [If they do, feel free to force push.]

ntalluri

I was looking over the current state of this dataset to help with the write-up for the registered report. As I looked over it, I left a few broad questions and reminders where what I was seeing didn’t fully match my understanding of the current planned approach for the dataset. This was not a full, in-depth review.

ntalluri · 2026-02-02T16:45:50Z

datasets/synthetic-data/scripts/panther_spras_formatting.py

+    edges_df = edges_df.sort_values(by=["NODE1", "NODE2", "Direction"], ascending=True, ignore_index=True)
+    edges_df = edges_df.drop_duplicates(keep="first", ignore_index=True)
+
+    edges_df.to_csv(out_folder / f"{pathway}_gs_edges.txt", sep="\t", index=False, header=False)


Reminder: This should be updated to make a gold standard of edges that are contained in both the pathway and the specific interactome.

ntalluri · 2026-02-02T16:46:31Z

datasets/synthetic-data/scripts/panther_spras_formatting.py

+    prizes_df["dummy"] = ""
+    prizes_df.rename(columns={"uniprot": "NODEID", "prizes": "prize"}, inplace=True)
+    result_df = prizes_df[["NODEID", "prize", "sources", "targets", "active", "dummy"]]
+    result_df.to_csv(out_folder / f"{pathway}_node_prizes.txt", sep="\t", index=False, header=True)


Reminder: This should be updated to make the input nodes to be those that are contained in both the pathway and the specific interactome.

datasets/synthetic-data/README.md

datasets/synthetic-data/scripts/sampling.py

datasets/synthetic-data/Snakefile

ntalluri · 2026-02-02T16:57:10Z

datasets/synthetic-data/scripts/sampling.py

+    pathway_df = convert_undirected_to_directed(pathway_df)
+    pathway_df = pathway_df[['Interactor1', 'Interactor2']]
+
+    print(f'Merging {pathway} with interactome...')


I'm assuming this isn't finished, but will you be testing for the properties that X% of the pathway edges are reachable and that X% of the pathway is in one connected component after merging the pathway into the sampled interactome?

ntalluri · 2026-02-02T17:01:05Z

datasets/synthetic-data/scripts/sampling.py

+    interactome_df = interactome_df.iloc[list(full_set)]
+
+    pathway_df = pandas.read_csv(
+        current_directory / '..' / 'processed' / pathway / f'{pathway}_gs_edges.txt', sep='\t',


I'm assuming this isn't finished, but will you be updating the gold standards once we have the downsampled interactomes. I’m assuming we’ll have a separate gold standard for each interactome for each pathway. The current '{pathway}_gs_edges.txt' is all of the edges in a pathway; some may not be reachable in an interactome.

tristan-f-r · 2026-02-02T17:24:18Z

This isn't finished. I'm going to run this for longer, but if you see the pathway.txt file generation, I haven't been able to get a connected s/t component after a concerning amount of runs.

[As for your gold standard comments, I'm specifically just worried about the interactome generation, and, if I can't get a valid sampled gs after 100 runs, I'm going to further optimize interactome sampling - gold standard thresholding can be easily finished with the inner join in sampling.py]

datasets/synthetic-data/README.md

tristan-f-r · 2026-02-02T17:49:05Z

I'll use that updated TF list 👍

for biopax -> extended sif conversion

tristan-f-r · 2026-02-11T23:43:24Z

As a textual tl;dr, I need to still add trim.py, and I want to update the paxtools utility to [efficiently] do BioPAX extraction from the large .owl file provided by PC.

I'm going to make that trim utility, split this PR to add those utilities, and merge the EGFR interactome updates and sampling using those utility scripts, then jump back to doing the BioPAX parsing.

ntalluri · 2026-02-12T16:13:40Z

datasets/synthetic-data/scripts/sampling.py

Would you be able to add to the README more information on how we are downsampling the interactomes + add the psuedo code?

ntalluri · 2026-02-12T16:17:15Z

datasets/synthetic-data/explore/README.md

Why are we providing this functionality?

As mentioned in the comment above, I want to stop downloading PathwayCommons SIF files individually and extract them from OWL. I originally added this file as a quick exploration tool of this data, but I'll drop it once you have your list of signaling pathways from PathwayCommons and the criteria you used to fetch them, and move it over to use said automated selection.

feat: synthetic pathways

20b1580

Co-Authored-By: Neha Talluri <78840540+ntalluri@users.noreply.github.com> Co-Authored-By: Oliver Faulkner Anderson <112665860+oliverfanderson@users.noreply.github.com> Co-Authored-By: Altaf Barelvi <altafayyubibarelvi@gmail.com>

tristan-f-r added the dataset Mutating datasets in any way. label Jul 1, 2025

ntalluri reviewed Jul 1, 2025

View reviewed changes

datasets/synthetic-data/Snakefile Outdated Show resolved Hide resolved

Merge branch 'main' into synthetic

fc12b4e

tristan-f-r mentioned this pull request Jul 30, 2025

dataset: DepMap #41

Merged

tristan-f-r and others added 5 commits January 5, 2026 19:03

Merge branch 'main' into synthetic

8ff381f

fix: use full protein links to unify synthetic with databases

f7c0c2d

Merge branch 'main' into synthetic

73b6d93

re-correct links

2ce621a

fix: interactome fetching

280b92a

tristan-f-r and others added 9 commits January 23, 2026 21:19

fix(diseases): fetch correct string links

db30556

chore: mv to scripts

0658528

chore: move to scripts, Pathify

e024e2c

style: fmt

7b09381

drop old thresholding

2a5feec

begin sampling

e389b32

chore: mv

af0ac30

rename

d1ade54

fix: compute weight counts normally

7483eea

tristan-f-r and others added 2 commits February 1, 2026 07:03

feat: weight-preserving sampling

05cf6d6

feat: sampling

58e9717

ntalluri reviewed Feb 2, 2026

View reviewed changes

datasets/synthetic-data/README.md Outdated Show resolved Hide resolved

ntalluri reviewed Feb 2, 2026

View reviewed changes

datasets/synthetic-data/README.md Outdated Show resolved Hide resolved

tristan-f-r and others added 14 commits February 2, 2026 20:38

feat: scripted sampling

a0f7079

chore: del some raw

3bb00e8

drop all raw interactome files

775d144

feat: finish up tf mapping again

5771bc7

feat: sampling on a pathway

813235d

style: fmt

7fb4642

chore: drop p38 mapk, add notes

83fee81

init candidates explorer

d7da699

chore: update directory urls

d45ec82

chore: drop all downloaded pathways

0d3b77e

fix: file extensions and such

751a8f2

chore: explore and such

2fceaa9

feat: base thresholding workflow

ac5b93c

chore: add paxtools

5cb7352

for biopax -> extended sif conversion

tristan-f-r added 2 commits February 12, 2026 01:09

feat: trimming

9d3e194

style: fmt

81a4e4e

ntalluri reviewed Feb 12, 2026

View reviewed changes

		@@ -0,0 +1,100 @@
		pathways = ["Apoptosis_signaling", "B_cell_activation",

Conversation

tristan-f-r commented Jul 1, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

ntalluri Jul 1, 2025

Choose a reason for hiding this comment

Uh oh!

Uh oh!

ntalluri commented Jul 3, 2025

Uh oh!

tristan-f-r commented Jul 28, 2025

Uh oh!

ntalluri commented Jul 28, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

ntalluri commented Jul 28, 2025

Uh oh!

ntalluri commented Jul 31, 2025

Uh oh!

ntalluri commented Aug 11, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

ntalluri commented Aug 13, 2025

Uh oh!

ntalluri commented Nov 6, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

ntalluri commented Nov 6, 2025

Uh oh!

tristan-f-r commented Jan 24, 2026

Uh oh!

ntalluri left a comment

Choose a reason for hiding this comment

Uh oh!

ntalluri Feb 2, 2026

Choose a reason for hiding this comment

Uh oh!

ntalluri Feb 2, 2026

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

ntalluri Feb 2, 2026

Choose a reason for hiding this comment

Uh oh!

ntalluri Feb 2, 2026

Choose a reason for hiding this comment

Uh oh!

tristan-f-r commented Feb 2, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

Uh oh!

tristan-f-r commented Feb 2, 2026

Uh oh!

tristan-f-r commented Feb 11, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

ntalluri Feb 12, 2026

Choose a reason for hiding this comment

Uh oh!

ntalluri Feb 12, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

tristan-f-r Feb 12, 2026

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

tristan-f-r commented Jul 1, 2025 •

edited

Loading

ntalluri commented Jul 28, 2025 •

edited

Loading

ntalluri commented Aug 11, 2025 •

edited

Loading

ntalluri commented Nov 6, 2025 •

edited

Loading

tristan-f-r commented Feb 2, 2026 •

edited

Loading

tristan-f-r commented Feb 11, 2026 •

edited

Loading

ntalluri Feb 12, 2026 •

edited

Loading