Speed up SimpleCov.collate with optional multi-process merge

[danielwestendorf]

This is an AI-Assisted PR

Collating the resultsets from a large CI matrix is slow in a way that scales badly. The collating process reads, parses and folds every shard in sequence, and nearly all the wall clock goes into that fold. This PR let's this process run on more than one core, speeding up overall collation.

Tested on real CI 1 process vs multi-processes (CI node cpu count = 4) for 160 real resultsets on a 231k line rails app saw collate reduce from 49sec to ~30sec.

┌────────────────────────┬─────────────┬──────┐
│                        │ merge time  │      │
├────────────────────────┼─────────────┼──────┤
│ single process         │ 49s         │      │
├────────────────────────┼─────────────┼──────┤
│ processes: 4           │ 30s         │ −61% │
├────────────────────────┼─────────────┼──────┤
│ processes: 6           │ 28s         │ −77% │
└────────────────────────┴─────────────┴──────┘

The API change

SimpleCov.collate takes a new processes: argument:

SimpleCov.collate Dir["simplecov-resultset-*/.resultset.json"], processes: 6

It defaults to 1, which never forks, so existing calls are completely unaffected — same code path as before. Everything else about the signature is unchanged.

Early attempts used Etc.nprocessors to automatically tune process count, however, this often reports the number of processes of the host machine, not the vm/container of which it runs, so I opted for manual tuning. Setting ENV var SIMPLECOV_CONCURRENCY will also work.

Failure handling

Every failure path returns nil rather than a partial merge, and the caller redoes the fold in-process. Reporting coverage for a subset of the resultsets would silently understate it, which is a worse outcome than being slow. That covers:

• a runtime that cannot fork — JRuby, TruffleRuby, Windows. Detected by the NotImplementedError the call raises, not by respond_to?, since those runtimes define Process.fork and only raise when it's invoked
• a worker that died
• a payload that came back truncated

Workers ship their folded pair back over a pipe, deserialized on a thread per worker so every pipe is drained while the workers are still writing — otherwise a payload larger than the pipe buffer blocks its worker mid-write and the parent blocks reaping a worker that can never finish. Children end with exit! so they never fall through to the collating process's at_exit handlers. Each worker folds its slice one file at a time, as the serial path does, so memory scales with the worker count rather than the resultset count.

[Copilot]

🟡 Changes recommended

A few updated/new comments reference a non-existent SimpleCov.parallel_collate, and processes: should be coerced/validated to an Integer early to avoid runtime errors from non-integer inputs.

Once you've addressed the issues Copilot identified, you can request another Copilot review.

This review doesn't count toward merge requirements. Sign up for the private preview to control whether Copilot approvals count.

Pull request overview

This PR adds an optional multi-process merge path for SimpleCov.collate, allowing large sets of resultset shards (e.g., CI matrices) to be merged faster by fanning out the fold across forked workers while preserving identical merge order and results.

Changes:

• Add processes: keyword to SimpleCov.collate (defaulting from SIMPLECOV_CONCURRENCY) and route merging through a new SimpleCov::ParallelResultMerger when processes > 1.
• Refactor ResultMerger to extract merge_resultsets for reuse by the parallel implementation.
• Add extensive specs, documentation, and benchmark harness updates to validate correctness and measure performance.

File summaries

File	Description
test_projects/faked_project/Rakefile	Adds a rake task exercising collate(..., processes: 2) in the fake project.
spec/support/merge_reference.rb	Adds a reference/oracle implementation of N-way merging for differential testing.
spec/support/merge_fuzzer.rb	Adds deterministic fuzz shard generation to hit adversarial merge edge cases.
spec/simplecov_spec.rb	Adds .collate across processes specs for equivalence and env/default behavior.
spec/result_merger_spec.rb	Adds unit coverage for ResultMerger.merge_resultsets and immutability of input paths.
spec/parallel_result_merger_spec.rb	Adds thorough specs for fan-out behavior, worker failure handling, chunking, etc.
spec/helper.rb	Introduces FORK_SUPPORTED and adjusts coverage filtering for fork-unreachable code paths.
spec/combine_differential_spec.rb	Adds end-to-end differential merge spec comparing fold vs reference rules.
sig/simplecov.rbs	Updates RBS signature/docs for SimpleCov.collate and adds merge_collated.
sig/internal/simplecov/result_merger.rbs	Adds RBS signature for new merge_resultsets helper.
sig/internal/simplecov/parallel_result_merger.rbs	Adds RBS definitions for the new parallel merger module.
README.md	Documents processes: usage, semantics, and fallback behavior.
lib/simplecov/result_processing.rb	Adds processes: to collate and routes through merge_collated.
lib/simplecov/result_merger.rb	Extracts merge_resultsets from merge_results for reuse and testing.
lib/simplecov/parallel_result_merger.rb	Introduces worker fan-out/collection implementation for parallel merges.
lib/simplecov.rb	Requires the new parallel_result_merger implementation.
features/test_unit_parallel_collate.feature	Adds cucumber feature asserting parallel collate produces identical output.
CHANGELOG.md	Documents the new processes: option, behavior, and benchmark results.
benchmarks/collate/runner.rb	Adds processes option and uses parallel merge in the merge phase when enabled.
benchmarks/collate/report.rb	Reports whether merge ran serially vs across forked workers; persists processes.
benchmarks/collate/cli.rb	Adds PROCESSES parsing and wiring into benchmark options.
benchmarks/collate.rb	Documents the new PROCESSES environment variable for the benchmark harness.

Review details

• Files reviewed: 22/22 changed files
• Comments generated: 4
• Review effort level: Lite

We're testing this review assessment. Please use 👍 or 👎 to tell us if it's correct.

[sferik]

Thanks for this PR! Overall, it looks quite good.

I agree with the first three Copilot comments. I’m not sure it’s necessary to coerce processes to an Integer (the fourth comment from Copilot), so feel free to skip that one.

Beyond what Copilot caught:

1. The differential test never exercises the parallel association. The "identical, not merely equivalent" claim depends on the combiners being associative: serial computes ((a+b)+c)+d while the fan-out computes (a+b)+(c+d). Same visiting order, different grouping. combine_differential_spec.rb only compares the serial :fold against the reference oracle, and the strategy parameter in mismatch_for is only ever called with :fold, which suggests a chunked strategy existed at some point and was dropped. The "same pair however many processes" spec uses simple line-only fixtures, so none of the adversarial reconcile_synthesized or drifted-key cases the fuzzer generates ever go through the chunked path. Adding a :chunked strategy (chunk, fold each slice, fold the slices) would directly pin the property the whole PR rests on with relatively little code.

2. The succeeded?(-1) spec is fragile. Process.wait2(-1) doesn't mean "a pid I can't reap", it means "wait for any child". If an unrelated child process is alive when that example runs, the call can block or reap a stranger's child and return true. It only passes today because nothing else leaves children around. A guaranteed-nonexistent positive pid (or stubbing wait2 to raise Errno::ECHILD) would make it more deterministic.

3. Naming nit: merge_fuzzer.rb and merge_reference.rb both say they serve spec/merge_differential_spec.rb, but the actual file is spec/combine_differential_spec.rb.

The PR description says non-fork runtimes are detected by the NotImplementedError the call raises rather than by respond_to?, but the code does the opposite (correctly, and the comment explains why). Also, the table’s percentages don’t match its own numbers (49s to 30s is about a 39% reduction, not 61%, and 49s to 28s is about 43%, not 77%). Mostly pointing this out for posterity, in case anyone ever looks back at this PR while researching or debugging a future issue.

One final thought: if fork raises EAGAIN partway through the fan-out, the already-spawned workers are never reaped and their readers leak. I agree with propagating the error, but an ensure in fan_out could still close readers and reap what was spawned before re-raising.

Happy to merge once the these issues are sorted.