# Evaluating time series forecasting models: An empirical study on performance estimation methods

arXiv: https://arxiv.org/abs/1905.11744 · HTML: https://mlsanity.com/paper/1905.11744/

## Key facts

- Cited by 53 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 28 May 2019.

## Built on by (most cited first, 30 of 53)

- [Forecast Evaluation for Data Scientists: Common Pitfalls and Best Practices](https://mlsanity.com/paper/2203.10716/) (2022)
- [Long-Range Transformers for Dynamic Spatiotemporal Forecasting](https://mlsanity.com/paper/2109.12218/) (2021)
- [How to avoid machine learning pitfalls: a guide for academic researchers](https://mlsanity.com/paper/2108.02497/) (2021)
- [LightAutoML: AutoML Solution for a Large Financial Services Ecosystem](https://mlsanity.com/paper/2109.01528/) (2021)
- [Active learning for data streams: a survey](https://mlsanity.com/paper/2302.08893/) (2023)
- [How to Evaluate Uncertainty Estimates in Machine Learning for Regression?](https://mlsanity.com/paper/2106.03395/) (2021)
- [DC-Check: A Data-Centric AI checklist to guide the development of reliable machine learning systems](https://mlsanity.com/paper/2211.05764/) (2022)
- [On Leakage in Machine Learning Pipelines](https://mlsanity.com/paper/2311.04179/) (2023)
- [Midwifery Learning and Forecasting: Predicting Content Demand with User-Generated Logs](https://mlsanity.com/paper/2107.02480/) (2021)
- [Time Series Analysis and Modeling to Forecast: a Survey](https://mlsanity.com/paper/2104.00164/) (2021)
- [Handling Concept Drift in Global Time Series Forecasting](https://mlsanity.com/paper/2304.01512/) (2023)
- [Evaluating Model Performance in Medical Datasets Over Time](https://mlsanity.com/paper/2305.13426/) (2023)
- [Model Selection for Time Series Forecasting: Empirical Analysis of Different Estimators](https://mlsanity.com/paper/2104.00584/) (2021)
- [Constrained or Unconstrained? Neural-Network-Based Equation Discovery from Data](https://mlsanity.com/paper/2406.02581/) (2024)
- [Modelling the COVID-19 virus evolution with Incremental Machine Learning](https://mlsanity.com/paper/2104.09325/) (2021)
- [Ensemble Modeling for Time Series Forecasting: an Adaptive Robust Optimization Approach](https://mlsanity.com/paper/2304.04308/) (2023)
- [Share Secrets for Privacy: Confidential Forecasting with Vertical Federated Learning](https://mlsanity.com/paper/2405.20761/) (2024)
- [Adaptive Anomaly Detection for Identifying Attacks in Cyber-Physical Systems: A Systematic Literature Review](https://mlsanity.com/paper/2411.14278/) (2024)
- [Simple Feedfoward Neural Networks are Almost All You Need for Time Series Forecasting](https://mlsanity.com/paper/2503.23621/) (2025)
- [Interpretable Machine Learning for Predicting Startup Funding, Patenting, and Exits](https://mlsanity.com/paper/2510.09465/) (2025)
- [Deep Similarity Learning for Sports Team Ranking](https://mlsanity.com/paper/2103.13736/) (2021)
- [Model Compression for Dynamic Forecast Combination](https://mlsanity.com/paper/2104.01830/) (2021)
- [Beyond Average Performance -- exploring regions of deviating performance for black box classification models](https://mlsanity.com/paper/2109.08216/) (2021)
- [Exceedance Probability Forecasting via Regression for Significant Wave Height Prediction](https://mlsanity.com/paper/2206.09821/) (2022)
- [Investigating Poor Performance Regions of Black Boxes: LIME-based Exploration in Sepsis Detection](https://mlsanity.com/paper/2306.12507/) (2023)
- [AALF: Almost Always Linear Forecasting](https://mlsanity.com/paper/2409.10142/) (2024)
- [Measuring temporal effects of agent knowledge by date-controlled tool use](https://mlsanity.com/paper/2503.04188/) (2025)
- [Towards Accurate Forecasting of Renewable Energy : Building Datasets and Benchmarking Machine Learning Models for Solar and Wind Power in France](https://mlsanity.com/paper/2504.16100/) (2025)
- [AIMM: An AI-Driven Multimodal Framework for Detecting Social-Media-Influenced Stock Market Manipulation](https://mlsanity.com/paper/2512.16103/) (2025)
- [Rolling-Origin Validation Reverses Model Rankings in Multi-Step PM10 Forecasting: XGBoost, SARIMA, and Persistence](https://mlsanity.com/paper/2603.20315/) (2026)

## Abstract (arXiv)

Performance estimation aims at estimating the loss that a predictive model will incur on unseen data. These procedures are part of the pipeline in every machine learning project and are used for assessing the overall generalisation ability of predictive models. In this paper we address the application of these methods to time series forecasting tasks. For independent and identically distributed data the most common approach is cross-validation. However, the dependency among observations in time series raises some caveats about the most appropriate way to estimate performance in this type of data and currently there is no settled way to do so. We compare different variants of cross-validation and of out-of-sample approaches using two case studies: One with 62 real-world time series and another with three synthetic time series. Results show noticeable differences in the performance estimation methods in the two scenarios. In particular, empirical experiments suggest that cross-validation approaches can be applied to stationary time series. However, in real-world scenarios, when different sources of non-stationary variation are at play, the most accurate estimates are produced by out-of-sample methods that preserve the temporal order of observations.