[Fix] Separate pytorch/jax build and test sub-jobs per phase

[HereThereBeDragons]

Summary

Quartz doubled pytorch/jax build cells and flipped a green build to
cancelled/failed whenever a nested test job failed (#82). The build leaf was
absorbing the run's nested test sub-jobs as if they were extra build cells. We
now partition a shared run's job list by phase, so the build leaf holds only
build sub-jobs and the test leaf only test sub-jobs. We also finalize the build
leaf early from a test-phase snapshot, so it never stalls in_progress waiting
for the run-level completion notify.

Fixes #82

Root cause

pytorch/jax build workflows invoke their test coverage as a reusable
workflow_call, so build and test jobs share one run id and one job list.
rockrel wraps each job in an uppercase ancestor segment
(Release | py 3.13 | JAX 0.11.0 / ...). A test-only cell borrowed that
ancestor's looser ref (0.11.0) while the sibling build cell carried its own
tail ref (rocm-jaxlib-v0.11.0), so the two landed as separate cells --
doubling the build leaf -- and the nested test's cancellation flipped the
build's rolled-up status.

What changed

• Partition signal. A test sub-job carries a Test | <arch> segment of its
  own; a build sub-job never does. _is_test_subjob encodes this, and
  _variants_from_jobs takes a phase argument to keep only the matching half.
  _TEST_ARCH_JOB_RE is now case-insensitive (matching _MATRIX_JOB_RE),
  because it is now the build-vs-test signal, not just arch extraction.
• Build leaf excludes nested test sub-jobs, so a cancelled or failed test
  can no longer flip a green build or add a phantom cell.
• Early build finalize (_refresh_same_run_build_from_test). When a
  test-phase notify arrives for a shared run, we scan the same job list for the
  build sub-jobs and finalize the build leaf, so it does not stall
  in_progress until the run-level completion notify fires.
• Renamed the same-run refreshers to a directional pair:
  _refresh_same_run_tests_from_build and _refresh_same_run_build_from_test.

Test plan

• Retargeted 3 tests that encoded the old build-absorbs-test behavior.
• Added tests for the JAX runs are not counted correctly #82 rockrel shape, the early build finalize, and
  standalone-test isolation (a standalone test dispatch has its own run id
  and must never rewrite the real build run's leaf).
• pytest scripts/receive_therock/tests/therock_update_status_json_test.py
  -> 116 passed.
• Full scripts/receive_therock/tests/ -> 429 passed + 16 subtests. The 3
  failures in therock_workflow_registry_test.py are pre-existing and
  unrelated (they scan real TheRock workflow YAML for notify_quartz
  phases).

[erman-gurses]

Please see my comments:

[erman-gurses]

Based on the test in here, what if, for example, 802 finishes earlier than 801

Quartz/scripts/receive_therock/tests/therock_update_status_json_test.py

Lines 1706 to 1722 in ebce36a

	def test_build_variants_merge_across_runs_into_platform_leaf() -> None:
	# Each (py, torch) cell is its own run; the platform build leaf aggregates
	# them, carrying only a rolled-up status plus the per-run variants.
	doc = StatusDocument()
	for rid, py in ((801, "3.10"), (802, "3.12")):
	run = _variant_run(
	pipeline_type="pytorch",
	pipeline_phase="build",
	run_id=rid,
	jobs=[_job(f"Build | py {py} | torch release/2.10")],
	)
	assert doc.upsert_leaf("linux", "", "pytorch", "build", tusj._create_leaf(run))
	build = doc.pipelines.pytorch.build["linux"]
	assert {v.run_id for v in build.variants} == {801, 802}
	assert {v.matrix["py"] for v in build.variants} == {"3.10", "3.12"}
	assert build.status is Status.success

Looks like it will be rejected.

Quartz/scripts/receive_therock/therock_status_document.py

Lines 241 to 263 in 4eca82c

	def should_replace(self, new: "RunLeaf") -> bool:
	"""Guard for a leaf slot: newest run wins, do-not-downgrade within a run.
	Consequence worth knowing: because run_id dominates run_attempt, a
	deliberate re-run of an *older* run (lower id) loses to an already-
	dispatched newer run (higher id) even at a higher attempt, and is
	silently dropped.
	"""
	if (
	self.run_id is not None
	and new.run_id is not None
	and new.run_id != self.run_id
	):
	return new.run_id > self.run_id
	existing_attempt = self.run_attempt or 0
	new_attempt = new.run_attempt or 0
	if new_attempt != existing_attempt:
	return new_attempt > existing_attempt
	if self.is_terminal() and not new.is_terminal():
	return False
	if self.completed_at is not None and new.completed_at is None:
	return False
	return True

[HereThereBeDragons]

for rid, py in ((801, "3.10"), (802, "3.12")):

these are independent build variants: one for py 3.10 and one for py 3.12. as such they are always listed and the run id does not matter

[erman-gurses]

Please see agent comment regarding build/test partition bypass via input fallback:

The concern is: phase filtering may correctly remove all test jobs while deriving a build, but then this unconditional fallback can create a build variant from workflow inputs anyway. Will this bypass the new build-vs-test partitioning?

[HereThereBeDragons]

see my wonderful claude reply:

Good catch on the fallback being phase-blind. It is a latent gap, not a live one, so I documented the invariant in-code rather than restructuring the #82 fix.

The fallback (_variants_from_inputs) runs only when the job-sourced derivation returns empty. For a build phase that happens only before any build-cell job exists in the run. At that instant no test sub-job exists either, since tests need builds. So the whole-run status the fallback reads reflects only setup/build-start state, never a test outcome, and it cannot stamp a build cell with a test-influenced status. This holds across all three topologies: the portable pytorch build (started notify inside the build job, so the fallback never fires for build at all) and the jax/pytorch release workflows (started notify in setup_matrix, before build jobs exist). Once the build and test jobs materialize, the job-sourced partition takes over and replaces the cell by key.

I discussed this with Claude to confirm the reachability holds for the release setup-job topology too.

[erman-gurses]

LGTM