# Model selection with proper scoring rules on data sets of time series: prefer the mean scaled score

arXiv: https://arxiv.org/abs/2606.24715 · HTML: https://mlsanity.com/paper/2606.24715/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 6 of its 23 references that are in the map.
- First posted on arXiv 23 Jun 2026.

## Builds on (most cited first, 6 of 6)

- [fev-bench: A Realistic Benchmark for Time Series Forecasting](https://mlsanity.com/paper/2509.26468/) (2025)
- [Forecast Evaluation for Data Scientists: Common Pitfalls and Best Practices](https://mlsanity.com/paper/2203.10716/) (2022)
- [Flexible Model Aggregation for Quantile Regression](https://mlsanity.com/paper/2103.00083/) (2021)
- [Local scale invariance and robustness of proper scoring rules](https://mlsanity.com/paper/1912.05642/) (2019)
- [Scalable Probabilistic Forecasting in Retail with Gradient Boosted Trees: A Practitioner's Approach](https://mlsanity.com/paper/2311.00993/) (2023)
- [Forecasting intermittent time series with Gaussian Processes and Tweedie likelihood](https://mlsanity.com/paper/2502.19086/) (2025)

## Built on by (most cited first, 1 of 1)

- [fable.intermittent: benchmarking probabilistic forecasting methods for intermittent time series](https://mlsanity.com/paper/2609.28607/) (2026)

## Abstract (arXiv)

We study the problem of model selection among probabilistic forecasting models evaluated on datasets of multiple time series. The performance of a model on a single time series is quantified by the average value (score) of a proper scoring rule over a test set, but extending model selection to data sets of time series requires aggregating these scores. Common approaches either rely on scaling scores and averaging them (mean scaled score) or avoid scaling by using alternative statistics such as mean ranks or win rates. However, these approaches can yield conflicting conclusions. We show that such discrepancies arise from the skewness of the distribution of the scores, which is particularly pronounced when test sets are short. The skewness can cause non-mean criteria (e.g., mean rank, median, win rate) to select misspecified models. In contrast, the mean score is immune from this problem. We further show that, as the size of the test sets increases, all aggregation criteria converge to the same model selection decision, mitigating these discrepancies. Our experiments on intermittent demand time series, including data from the M5 competition, highlight the importance of sufficiently large test sets; the mean scaled score appears to be the more reliable approach, also because empirically we found its decision to remain consistent when different scaling factors are adopted.