OpenSTEF Meta V0.1

[Lars800]

Introducting OpenSTEF-Meta, a one-stop-shop for all things meta learning.

This sub-package introduces four common meta learning algorithms:

Residual Forecaster

• 2 stage Forecasting Model
• Primary model is fitted as usual
• Secondary model fitted on residuals

Stacking Forecaster

• Multiple Base Forecasters
• Single Regressor is fitted on Base Predictions

Learned Weights Forecaster

• Multiple Base Forecasters
• Classification model learns optimal model weights

Rules Forecaster

• Multiple Base Forecasters
• Pre-defined rules on how to combine base methods.

---

This is an initial implementation

• The methods have been tested on the Liander 2024 Huggingface dataset.
• The results are accurate and improve over existing LGBM, XGBoost and GBLinear models
• The code can still be further optimized for efficiency

[egordm]

Awesome improvements! It looks mostly release-ready.

I have a few small comments / nitpicks, but overall great work!

Quick note, we did make some small changes/fixes in the current release branch. So you might need to rebase.

I think after this is merged, we can open a PR to merge the research branch into release if it's free of research artifacts / temporary scripts.

[egordm]

We should probably address this before merging.

It's mostly in hyperparams if I understand it correctly? Since the rest is pickling.

[Lars800]

Yes, this is required for use in production as I understand it. I left this integration un resolved as I do not fully understand this part of the package.

The primary issue here is EnsembleForecastingModel does not have a Forecaster attribute. So forecaster.hyperparams is unavailable. Depending on what they are used for downstream, we can either:

• Pass combiner.hyperparams
• Pass a dictionary of hyperparams, something like

{
'forecaster_model_1' : Hyperparams()
'forecaster_model_2' : Hyperparams()
'combiner' : Hyperparams()
}

This way everything is neatly saved, but it does require additional changes

[egordm]

Nitpick. But I think using log level info might be too verbose. debug would be more appropriate, or no logging at all, since it's not really actionable.

[Lars800]

I will change the log level. I do think it would be nice to keep the full EnsembleModelFit result in the future, to really evaluate the setup.

current setup:

EnsembleModelFitResults:
    forecaster_results: ModelFitResult[] // Performance of base forecasters
    combiner_results: ModelFitResult // Performance of forecasters + combiner

Right now the combiner fit result gives the total performance after applying the combiner models. Ideally we would separate this. We would get something like this:

EnsembleModelFitResults:
    forecaster_results: ModelFitResult[] // Performance of base forecasters
    combiner_results: ModelFitResult // Added performance of combiner 
    full_results: ModelFitResult // Performance of forecasters + combiner

This off course implies extra work for the callbacks etc to make everyting compatible, so for now we discard larger structure and keep only the total result.

[MvLieshout]

Do we also enforce this?

Then we do not have to assume I guess

[Lars800]

We do not enforce it explicitly, but if the Forecasting Workflow (Config) is used, it is always the case. I have removed the inline comment. If statement still applies, as ensemble forecasting workflow does not have a .forecaster property. Alternatively we can define a .forecaster property on Ensemble Forecasting Model that returns self.forecasters[0] (To ensure compatibility without If statements with beam etc.

[MvLieshout]

Really nice additions to OpenSTEF, I have left some comments. Mostly small nitpicks.