# A Look at the Evaluation Setup of the M5 Forecasting Competition

arXiv: https://arxiv.org/abs/2108.03588 · HTML: https://mlsanity.com/paper/2108.03588/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- Builds on 2 of its 20 references that are in the map.
- First posted on arXiv 8 Aug 2021.

## Builds on (most cited first, 2 of 2)

- [Performance Metrics (Error Measures) in Machine Learning Regression, Forecasting and Prognostics: Properties and Typology](https://mlsanity.com/paper/1809.03006/) (2018)
- [Sales Demand Forecast in E-commerce using a Long Short-Term Memory Neural Network Methodology](https://mlsanity.com/paper/1901.04028/) (2019)

## Built on by (most cited first, 4 of 4)

- [Model scale versus domain knowledge in statistical forecasting of chaotic systems](https://mlsanity.com/paper/2303.08011/) (2023)
- [Regions of Reliability in the Evaluation of Multivariate Probabilistic Forecasts](https://mlsanity.com/paper/2304.09836/) (2023)
- [Data-driven Modeling of Combined Sewer Systems for Urban Sustainability: An Empirical Evaluation](https://mlsanity.com/paper/2408.11619/) (2024)
- [Evaluation Choices Decide the Forecasting Leaderboard: Evidence from a Production Marketplace Panel](https://mlsanity.com/paper/2609.27867/) (2026)

## Abstract (arXiv)

Forecast evaluation plays a key role in how empirical evidence shapes the development of the discipline. Domain experts are interested in error measures relevant for their decision making needs. Such measures may produce unreliable results. Although reliability properties of several metrics have already been discussed, it has hardly been quantified in an objective way. We propose a measure named Rank Stability, which evaluates how much the rankings of an experiment differ in between similar datasets, when the models and errors are constant. We use this to study the evaluation setup of the M5. We find that the evaluation setup of the M5 is less reliable than other measures. The main drivers of instability are hierarchical aggregation and scaling. Price-weighting reduces the stability of all tested error measures. Scale normalization of the M5 error measure results in less stability than other scale-free errors. Hierarchical levels taken separately are less stable with more aggregation, and their combination is even less stable than individual levels. We also show positive tradeoffs of retaining aggregation importance without affecting stability. Aggregation and stability can be linked to the influence of much debated magic numbers. Many of our findings can be applied to general hierarchical forecast benchmarking.