Write UberGraph concords in a deterministic order

[gaurav]

Draft: the one-line change is here, but it has not been measured yet. See "What still needs doing".

Fixes #902.

The bug

src/ubergraph.py:build_sets() writes each subject's xref targets by iterating a Python set:

v = set([norm(x, other_prefixes) for x in v])
for x in v:                       # <- randomized string-hash order, per process
    ...
    concordfiles[p].write(f"{k}\t{types2relations[set_type]}\t{x}\n")

Set-of-strings iteration order is randomized per process, so the same UberGraph data produces differently-ordered concord files on every pull. This PR sorts before writing.

Why order changes cliques

glom() resolves a unique_prefixes conflict by keeping whichever bridge it processes first and silently dropping the other (src/babel_utils.py, the if not setok: continue branch). When a clique has two competing bridges to the same unique prefix, concord line order picks the winner.

Worked example from #886: MONDO:0003425 "ophthalmoplegia" reaches two different HP terms, and DISEASE_UNIQUE_PREFIXES admits only one HP:

• MONDO:0003425 --exactMatch--> UMLS:C0029089 --eq--> HP:0000602 "Ophthalmoplegia" (correct)
• MONDO:0003425 --exactMatch--> SNOMEDCT:78097002 --xref--> HP:0007824 "Total ophthalmoplegia" (a subtype; bad MONDO xref, worked around in Report MONDO:0003425 -> SNOMEDCT:78097002 bad exactMatch upstream to MONDO #900)

Different pulls of the identical MONDO concord put a different bridge first, and the clique flips. Confirmed by swapping just those two lines in the concord and re-glomming.

The blast radius is not small: 33 of the 93 rows in #886's clique diff were this noise rather than the change under test. Re-glomming that branch's own inputs with the MP source fully excluded reproduced all 33.

Scope: this fixes reproducibility, not correctness

#894 originally recorded this behavior as "deterministic given identical inputs ... not Python hash-seed nondeterminism" and "independent of concord order". Both were wrong, and that issue has been corrected and rescoped; the nondeterminism is now stated properly in #902, which this PR closes.

What #894 still tracks, and what this PR does not do, is make the tie-break correct. Sorting only makes an arbitrary rule reproducible: the winner becomes "alphabetically smallest target CURIE", which has nothing to do with evidence strength. In the example above SNOMEDCT:78097002 sorts before UMLS:C0029089, so after this PR the wrong HP wins by default and it takes a hand-curated input_data/mondo_badxrefs.txt entry (#900) to correct that one clique. Awarding contested cross-references by exactMatch > closeMatch > hasDbXref is the real fix.

That is an argument for landing #894's fix too, not for holding this one: today the winner is both arbitrary and random, and randomness is the part that corrupts impact reports.

What still needs doing before this leaves draft

• Rebuild disease/phenotype before and after this change and run babel-clique-diff; commit the artifacts under docs/pipelines/.
• Size the churn. Sorting picks a winner wherever a unique_prefixes conflict exists today, and per the above it may pick a worse one than the last pull happened to. Worth counting how many contested cliques exist and spot-checking whether the alphabetical winner is systematically wrong.
• Decide whether to land glom: unique_prefixes tie-break for a contested cross-reference ignores evidence strength #894's evidence-strength tie-break on top of this, or instead of it.
• Check the other build_sets() callers (anatomy, chemicals, gene/protein) for the same sensitivity.
• Consider logging glom()'s merge refusals, so contested bridges stop being invisible.

🤖 Generated with Claude Code