# Model Selection for Time Series Forecasting: Empirical Analysis of Different Estimators

arXiv: https://arxiv.org/abs/2104.00584 · HTML: https://mlsanity.com/paper/2104.00584/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Builds on 11 of its 58 references that are in the map.
- First posted on arXiv 1 Apr 2021.

## Builds on (most cited first, 11 of 11)

- [XGBoost: A Scalable Tree Boosting System](https://mlsanity.com/paper/1603.02754/) (2016)
- [DeepAR: Probabilistic Forecasting with Autoregressive Recurrent Networks](https://mlsanity.com/paper/1704.04110/) (2017)
- [Temporal Fusion Transformers for Interpretable Multi-horizon Time Series Forecasting](https://mlsanity.com/paper/1912.09363/) (2019)
- [N-BEATS: Neural basis expansion analysis for interpretable time series forecasting](https://mlsanity.com/paper/1905.10437/) (2019)
- [ranger: A Fast Implementation of Random Forests for High Dimensional Data in C++ and R](https://mlsanity.com/paper/1508.04409/) (2015)
- [Should we really use post-hoc tests based on mean-ranks?](https://mlsanity.com/paper/1505.02288/) (2015)
- [Evaluating time series forecasting models: An empirical study on performance estimation methods](https://mlsanity.com/paper/1905.11744/) (2019)
- [Improving the Accuracy of Global Forecasting Models using Time Series Data Augmentation](https://mlsanity.com/paper/2008.02663/) (2020)
- [Global Models for Time Series Forecasting: A Simulation Study](https://mlsanity.com/paper/2012.12485/) (2020)
- [Machine Learning vs Statistical Methods for Time Series Forecasting: Size Matters](https://mlsanity.com/paper/1909.13316/) (2019)
- [How to evaluate sentiment classifiers for Twitter time-ordered data?](https://mlsanity.com/paper/1803.05160/) (2018)

## Built on by (most cited first, 3 of 3)

- [Lag Selection for Univariate Time Series Forecasting using Deep Learning: An Empirical Study](https://mlsanity.com/paper/2405.11237/) (2024)
- [ModelRadar: Aspect-based Forecast Evaluation](https://mlsanity.com/paper/2504.00059/) (2025)
- [Forecasting with Deep Learning: Beyond Average of Average of Average Performance](https://mlsanity.com/paper/2406.16590/) (2024)

## Abstract (arXiv)

Evaluating predictive models is a crucial task in predictive analytics. This process is especially challenging with time series data where the observations show temporal dependencies. Several studies have analysed how different performance estimation methods compare with each other for approximating the true loss incurred by a given forecasting model. However, these studies do not address how the estimators behave for model selection: the ability to select the best solution among a set of alternatives. We address this issue and compare a set of estimation methods for model selection in time series forecasting tasks. We attempt to answer two main questions: (i) how often is the best possible model selected by the estimators; and (ii) what is the performance loss when it does not. We empirically found that the accuracy of the estimators for selecting the best solution is low, and the overall forecasting performance loss associated with the model selection process ranges from 1.2% to 2.3%. We also discovered that some factors, such as the sample size, are important in the relative performance of the estimators.