RFC: Performance, Memory, and Observability enhancements (Parallel matching, Caching, OOM prevention)

[stan-buren]

Context & TL;DR

While working on the EIC code integration in PR #306, I encountered a few local performance bottlenecks during testing. Specifically, the full pipeline took ~40-60 minutes per run, occasionally crashed with an Out-Of-Memory (OOM) error when processing the 3.9 GB MASTR Solar CSV alongside other datasets, and was somewhat hard to debug during silent processing phases.

To speed up my local development loop, I implemented several optimizations—caching, parallel processing, chunking, and logging.

I am opening this issue to share the results and ask the maintainers: would you be interested in seeing any of these contributed back to the main repository?

Local Benchmark (Before → After)

Metric	Original	With my local optimizations	How it was achieved
Full pipeline (8 sources)	~40 minutes	6 min (cold), 3.7 min (warm)	ProcessPoolExecutor for pairwise comparisons
Peak RAM (incl. MASTR)	OOM (~16 GB exceeded)	~2 GB	Chunked MASTR parsing & batch streaming
Repeated runs	Full re-download & parse	38% faster	Hash-based invalidation cache
Observability	Silent until completion/crash	Detailed	Progress logging per-source and per-country

Key insight: The raw data volume itself is quite small (~1.4 GB total on disk). The main bottleneck appears to be the sequential execution of Duke's O(N×M) comparisons across 45 separate JVM instances.

---

Identified Bottlenecks & Proposed Solutions

1. Memory limits (OOM) on MASTR

Current state: collect() loads datasets into memory concurrently via parmap. Loading MASTR (141K records, ~8GB memory footprint) alongside GEM and others can push memory usage past 16 GB, causing swap thrashing and OOM kills on standard developer machines.
Proposed fix:

• Implement chunked reading (chunksize=100_000) for large files in data/mastr.py.
• Optionally, stream the collect phase: process and save one source to disk, free memory, then load the next. This reduces peak RAM to just the size of the largest single source (~2 GB).

2. Sequential Duke execution

Current state: Duke runs Java subprocesses sequentially (one per country pair). 10 sources result in 45 pairwise comparisons, executed one by one with JVM startup overhead for each.
Proposed fix: Wrap the country-pair loop in matching.py with a ProcessPoolExecutor (defaulting to os.cpu_count()). Since each comparison handles its own isolated JVM lifecycle, they parallelize perfectly. In my tests, this drops the matching time from ~40 minutes down to ~6 minutes.

3. Cold starts on repeated runs

Current state: Running with update=True re-downloads and re-parses all data, even if only one downstream configuration or a single file was changed.
Proposed fix: A lightweight, dependency-free pickle cache in ~/.cache/powerplantmatching/source_cache/, invalidated by raw file metadata hashes (mtime + size). This bypasses the parsing phase entirely for unchanged sources during iterative development.

4. Observability

Current state: Long operations (like GEM loading 8 Excel files or Duke crunching cross-matches) run silently. If a JVM crashes, it can be hard to trace.
Proposed fix: Inject standard Python logging to emit progress updates (e.g., [ENTSOE × OPSD] match: 17/23 countries) and catch unhandled exceptions with full tracebacks before JVM teardowns hide them.

---

Note on Implementation

To test these changes locally without constantly breaking the upstream research code, I temporarily encapsulated them in an opt-in core/ module with a boolean toggle.

However, I do not propose merging a parallel core/ wrapper into this repository. That was just my local testing sandbox. If any of these features are desired, I would integrate them cleanly and directly into the existing architecture (matching.py, data.py, etc.).

---

Potential Future Optimization: Pre-filtering (Blocking)

(Just sharing an architectural observation, not pushing for immediate implementation)

Currently, the pipeline evaluates every record against every other record. For example, matching ENTSO-E against GEM results in roughly 17K × 23K = 391 million comparisons executed in a single-threaded Java loop.

A standard record linkage best practice is blocking—filtering out highly improbable pairs before running the expensive scoring algorithm. Implementing a fast, lightweight blocking step (e.g., using DuckDB to run a vectorized SQL query that filters by jaro_winkler_similarity > 0.7 and haversine distance) before passing the remaining candidates to Duke could reduce the workload by 100x to 1000x.

This would complement any future fuzzy matching engines discussed in PR #301.

---

Next Steps

There are a lot of ideas here, and I don't want to dump a massive, hard-to-review PR on the project. Instead, I would prefer to submit these as small, atomic PRs based on what the maintainers feel is most urgent.

Which of these would be most valuable to your workflow right now?

1. Parallel execution (~3 lines of code in matching.py, massive speedup)
2. Chunked MASTR parsing (Prevents OOM crashes)
3. Source caching (Speeds up iterative development)
4. Structured logging (Better debugging)

I have all of this working locally and can open focused PRs whenever convenient. Let me know what you think!

Related PRs:

• Fix EIC code aggregation and integrate JRC-PPDB-OPEN data source  #306 — EIC deterministic matching (where these bottlenecks were encountered)
• Replace Java DUKE with a pure-Python rapidfuzz matching engine #301 — Replace Java Duke with rapidfuzz

[stan-buren]

To add a bit more context to the metrics above, here is what these optimizations actually look like under the hood when running.

1. Hardware Utilization (Parallel Duke)
Here is a snapshot of htop during the heaviest cross-matching phase:

before

after

As you can see, all 8 CPU cores are fully utilized by the isolated JVM instances working in parallel. More importantly, memory consumption remains completely stable at ~5.4 GB, eliminating the OOM spikes entirely.

2. Structured Observability

Instead of minutes of silent processing, the pipeline now emits clear, real-time milestones. I recorded a quick screen capture of the logs in action:

Screen.Recording.2026-07-12.at.11.24.02.mov

You can see exactly which sources are being compared, how many workers are active, and the exact execution times for the bottleneck functions. It makes tracing crashes and monitoring progress significantly easier.