Fix EIC code aggregation and integrate JRC-PPDB-OPEN data source

[stan-buren]

🎯 TL;DR

This PR fixes a silent bug in reduce_matched_dataframe that caused 99.98% of EIC codes to be dropped from the final output. It also integrates the JRC-PPDB-OPEN dataset to provide missing coordinates for ENTSO-E records.

Addresses the data foundation for #287 (EIC-first matching). Complementary to #289 which implements the EIC-based matching pass

Metric	Before	After
Records with preserved EIC codes	19 / 165,064 (0.01%)	1,216 / 165,068 (0.7%)
ENTSOE plants with EIC + coordinates (JRC JOIN)	0	1,901

---

🐛 The Problem

Currently, the EIC column in powerplants.csv is populated almost entirely with {nan}. This happens due to two issues in matching.py:

1. Column name mismatch: In reduce_matched_dataframe(), the aggregation rule is defined as "eic_code": set, but the actual column defined in config.yaml is "EIC". The set aggregation is never applied.
2. Reliability score overwrite: Because the aggregation falls back to the default "first", sources with higher reliability scores (like GEM, score=6) overwrite ENTSO-E (score=5). Since GEM doesn't provide EIC codes, the valid ENTSO-E EICs are overwritten with NaN.

Additionally, because ENTSO-E 14.1.B does not publish exact coordinates, Duke's spatial matching drops ~93% of ENTSO-E records during the matching phase.

---

🛠️ The Solution

This PR fixes the EIC pipeline at the data level:
1. Fix EIC Aggregation (matching.py)
Changed "eic_code": set to "EIC": lambda x: set(...). EICs are now correctly collected from all matched sources into a set, rather than being overwritten by the highest-scored source. This alone recovers over 1,200 EIC codes for existing matched records.

2. JRC-PPDB-OPEN Integration (data.py)
To solve the lack of ENTSO-E coordinates, this PR integrates the JRC Open Power Plants Database (v1.0, European Commission).

• JRC-PPDB-OPEN links ENTSO-E eic_p to exact WGS84 coordinates.
• The JRC_PPDB_OPEN() function reads the dataset, aggregates generation units to production units, and produces a PPM-compatible DataFrame.
• Result: 1,901 European power plants now have accurate coordinates via a deterministic JOIN on eic_p.

3. Edge Case Guard (cleaning.py)
Added Capacity = Capacity.replace(0, pd.NA) to prevent division by zero in weighted capacity calculations.

🔄 Backward Compatibility

• Safe fallback: Sources without an EIC column are unaffected — the set(...) lambda handles missing values gracefully.
• Non-breaking: No API changes to powerplants() or collect(). The config update is purely additive.

📈 Why this matters for PyPSA

Preserving the EIC code in the final powerplants.csv is the missing bridge to operational ENTSO-E data. By having accurate EICs, the PyPSA-Earth community can now easily join the dataset with ENTSO-E 15.1 (Unavailability/Outages) and 16.1.A (Actual Generation) for advanced model validation and dynamic availability constraints.

[stan-buren]

📝 Research Diary: The Missing Bridge Between ENTSO-E and Spatial Data

For the maintainers: I wanted to share the background research that led to this PR. Finding the right way to handle EIC codes turned into quite a rabbit hole, and I tried several dead-end approaches before landing on the JRC-PPDB-OPEN integration. Here is the full context of what I found.

1. Attempt 1: The OpenStreetMap

I already knew that OSM data for power plants requires substantial curation to be usable at scale. However, I decided to dig into osm-powerplants and the raw tags anyway to see if there was some untapped potential we were missing. My theory was simple: let's just extract EIC codes directly from OSM.

I checked OSM Taginfo and found 4,765 objects tagged with ref:EU:ENTSOE_EIC (the uppercase variant). I thought this could be our silver bullet—we could bypass the curation mess, extract these tags, do a deterministic exact JOIN with ENTSO-E 14.1.B, and easily borrow the coordinates from OSM.

Why it failed:
When I ran the actual intersection between the OSM EIC tags and the ENTSO-E 14.1.B dataset, I found exactly 22 overlaps.
It turns out OSM volunteers primarily tag substations, minor generators, and historical sites. ENTSO-E 14.1.B, on the other hand, strictly tracks transmission-connected Production Units (mostly >100MW). They are two completely different universes. Relying on OSM tags alone wouldn't solve the matching problem.

2. Attempt 2: Brute-force Geocoding via Nominatim

Since ENTSO-E 14.1.B provides text-based location data, I tried geocoding it myself.
I downloaded the full FMS 14.1.B extract (39,242 records). I filtered out generic names (like just "Belgium") which left me with 21,170 records with detailed text locations. I wrote a script to feed them into Nominatim.

The result: I successfully geocoded ~92% (19,473 records).
Why I scrapped it: It was a terrible pipeline solution. Nominatim heavily rate-limits requests, 289 locations were completely unfound, and the accuracy was roughly "town-center" level, not the exact chimneys or dam walls. It was a fragile band-aid.

3. The Regulatory Rabbit Hole: Why doesn't ENTSO-E just publish coordinates?

Working with ENTSO-E data daily, I already knew they don't include exact lat/lon coordinates alongside their MW capacities in public extracts. However, I wanted to understand the root cause behind this architectural decision. It turns out, the absence of spatial data is a feature, not a bug, baked directly into European law:

• Regulation (EU) 543/2013 was designed strictly for market transparency (prices, bidding zones, capacities) to prevent insider trading. It is not a GIS tool.

• Directive 2008/114/EC focuses on the protection of critical infrastructure. Transmission System Operators (TSOs) can and actively do hide exact locations to prevent sabotage.

• When TSOs exchange data internally using the CGMES / IEC 62325 standards, the coordinates are included. However, they are explicitly filtered out before being published to the Transparency Platform. Even the official ENTSO-E Grid Map actively distorts node positions to prevent exact mapping.

4. The Breakthrough: JRC-PPDB-OPEN

If TSOs legally can't publish coordinates, how do we map ENTSO-E data?
I found the answer in the European Commission's Joint Research Centre (JRC). In 2019, JRC researchers encountered this exact same problem while trying to build energy models.

Since they couldn't force ENTSO-E to release coordinates, they built the JRC Open Power Plants Database (DOI: 10.5281/zenodo.3574566). They took the ENTSO-E registry and cross-referenced it with open academic and environmental registries (like WRI Powerwatch and GEO) that do have coordinates.

Because JRC is an academic body compiling open data, they aren't bound by the TSO security directives. Their dataset provides a clean, validated CSV mapping eic_p (Production Unit EIC) and eic_g (Generation Unit EIC) to exact WGS84 lat/lon coordinates.

By integrating this single CSV into data.py, we instantly give 1,901 major European power plants exact coordinates via a deterministic EIC join.

🚀 The Grand Vision: What EIC Preservation Unlocks for PyPSA

Currently, PyPSA-Earth uses PPM strictly for static installed capacity. But by fixing the EIC aggregation bug in this PR and ensuring every matched plant retains its authoritative EIC code, PPM becomes a gateway to operational, real-time ENTSO-E data.

In future PRs, this deterministic key allows to easily JOIN powerplants.csv with:

• 16.1.A (Actual Generation): To backtest PyPSA's dispatch optimization against historical reality.
• 15.1 (Unavailability): To replace flat, theoretical availability assumptions (e.g., "coal is available 85% of the time") with actual, per-plant historical Forced Outage Rates.
• ProductionAndGenerationUnits_r3: To perfectly aggregate Generation Units into Production Units, rather than relying on brittle groupby(["Name", "Fueltype"]) string matching.
• EU ETS / E-PRTR (via EUI Cadmus mappings): To assign verified, exact CO₂ and NOx emission constraints to specific nodes in the grid.

I hope this context helps explain why I tackled the EIC issue from this specific angle. Happy to discuss the architecture further!

[stan-buren]

Follow-up: Relationship to PRs #289 and #301

Two other open PRs are directly relevant to this work, and I wanted to flag how they intersect to ensure a smooth merge path:

PR #289 (@MaykThewessen) — EIC-first deterministic matching (_match_by_eic)
@MaykThewessen has implemented a highly robust EIC-based matching pass using a pandas explode+merge approach with degree-1 uniqueness checks. This is a brilliant solution because it correctly handles edge cases like the Alpine hydro scheme (where ENTSO-E aggregates under one EIC while OPSD splits into 8 stations—a simple shared-EIC match would incorrectly pair the aggregate with whichever station it hits first).

The key finding from #289 is that Duke fuzzy matching currently misses 55 out of 371 EIC-confirmed pairs (34.6 GW of capacity).

Our PRs are strictly complementary, not overlapping:

• Add EIC-first deterministic matching before Duke fuzzy matching #289 provides the robust matching algorithm (_match_by_eic).
• This PR (Fix EIC code aggregation and integrate JRC-PPDB-OPEN data source  #306) provides the required data foundation. Without the "eic_code"→"EIC" fix in this PR, _match_by_eic has nothing to match against because 99.98% of EICs are wiped out and turn into {nan} during reduction. Additionally, this PR introduces the JRC-PPDB-OPEN source, providing exact coordinates for 1,901 ENTSO-E plants so Duke can process the fallbacks correctly.

Suggested merge flow: Preserve EICs via #306 → Match deterministically via #289 → Join coordinates via JRC.

PR #301 (@FabianHofmann) — Replace Duke with rapidfuzz
The move from Java-based Duke to pure-Python rapidfuzz (~17× faster) is completely orthogonal to both EIC PRs. In fact, an EIC-first deterministic matching pass significantly reduces the workload for whichever fuzzy engine is ultimately used, since exact EIC-matched pairs are removed from the pool before fuzzy matching even runs.

Happy to rebase or restructure if the maintainers prefer a different split between these updates!

[MaykThewessen]

Sounds good!