Summary
The MaStR data source is frozen to a single Zenodo release, and the loader additionally hardcodes that release's date inside the zip, so even bumping the config to a newer dump is not enough to make it load.
1. The source is pinned to one Zenodo dump
config.yaml pins:
MASTR:
fn: bnetza_open_mastr_2025-02-09.zip
url: https://zenodo.org/records/14783581/files/bnetza_open_mastr_2025-02-09.zip
German plant data therefore only refreshes when a new dump is published to Zenodo, which happens infrequently. Between releases every powerplantmatching user gets stale German generation/capacity, with no in-library way to use a more recent Marktstammdatenregister export.
2. The loader hardcodes the dump's internal folder date
In MASTR() (data.py ~L2411) the storage-units CSV is read from a fully hardcoded path that embeds the release date:
fn_storage_units = (
"bnetza_open_mastr_2025-02-09/bnetza_mastr_storage_units_raw.csv"
)
storage_units = pd.read_csv(file.open(fn_storage_units), usecols=cols)The moment MASTR.fn/url is bumped to a newer dump (whose inner folder is bnetza_open_mastr_<newer-date>/), this read fails with a KeyError. So updating the config alone is not sufficient; a code edit is required for every new release.
A related schema drift: ThermischeNutzleistung is present in the Zenodo CSV dump but absent from the open_mastr bulk export, so any non-Zenodo-shaped dump also breaks the CHP-detection assign that references that column.
Proposed fixes
- Date-agnostic inner path. Match the storage-units CSV by suffix (
name.endswith("bnetza_mastr_storage_units_raw.csv")) instead of a hardcoded dated folder, so any dated dump loads unchanged.
- Schema-tolerant CHP detection. Fall back to
KwkMastrNummer alone when ThermischeNutzleistung is absent.
- Glob-capable
fn. Allow MASTR.fn to be a glob (bnetza_open_mastr_*.zip) so a locally built, dated dump is auto-selected (newest wins; ISO dates sort chronologically), with the frozen Zenodo url kept only as a download seed.
Items 1-3 are small and behavior-preserving on the current Zenodo dump. I have these working locally and will open a PR. A fourth, optional piece (a helper script that rebuilds a current dump from the open_mastr bulk export, decoupling freshness from the Zenodo cadence) can follow separately if there is interest.
Summary
The MaStR data source is frozen to a single Zenodo release, and the loader additionally hardcodes that release's date inside the zip, so even bumping the config to a newer dump is not enough to make it load.
1. The source is pinned to one Zenodo dump
config.yamlpins:German plant data therefore only refreshes when a new dump is published to Zenodo, which happens infrequently. Between releases every
powerplantmatchinguser gets stale German generation/capacity, with no in-library way to use a more recent Marktstammdatenregister export.2. The loader hardcodes the dump's internal folder date
In
MASTR()(data.py~L2411) the storage-units CSV is read from a fully hardcoded path that embeds the release date:The moment
MASTR.fn/urlis bumped to a newer dump (whose inner folder isbnetza_open_mastr_<newer-date>/), this read fails with aKeyError. So updating the config alone is not sufficient; a code edit is required for every new release.A related schema drift:
ThermischeNutzleistungis present in the Zenodo CSV dump but absent from theopen_mastrbulk export, so any non-Zenodo-shaped dump also breaks the CHP-detection assign that references that column.Proposed fixes
name.endswith("bnetza_mastr_storage_units_raw.csv")) instead of a hardcoded dated folder, so any dated dump loads unchanged.KwkMastrNummeralone whenThermischeNutzleistungis absent.fn. AllowMASTR.fnto be a glob (bnetza_open_mastr_*.zip) so a locally built, dated dump is auto-selected (newest wins; ISO dates sort chronologically), with the frozen Zenodourlkept only as a download seed.Items 1-3 are small and behavior-preserving on the current Zenodo dump. I have these working locally and will open a PR. A fourth, optional piece (a helper script that rebuilds a current dump from the
open_mastrbulk export, decoupling freshness from the Zenodo cadence) can follow separately if there is interest.