# Forecast Evaluation for Data Scientists: Common Pitfalls and Best Practices

arXiv: https://arxiv.org/abs/2203.10716 · HTML: https://mlsanity.com/paper/2203.10716/

## Key facts

- Cited by 55 papers in the map (data through 14 Oct 2026).
- Builds on 28 of its 128 references that are in the map.
- First posted on arXiv 21 Mar 2022.

## Builds on (most cited first, 28 of 28)

- [Informer: Beyond Efficient Transformer for Long Sequence Time-Series Forecasting](https://mlsanity.com/paper/2012.07436/) (2020)
- [Autoformer: Decomposition Transformers with Auto-Correlation for Long-Term Series Forecasting](https://mlsanity.com/paper/2106.13008/) (2021)
- [FEDformer: Frequency Enhanced Decomposed Transformer for Long-term Series Forecasting](https://mlsanity.com/paper/2201.12740/) (2022)
- [DeepAR: Probabilistic Forecasting with Autoregressive Recurrent Networks](https://mlsanity.com/paper/1704.04110/) (2017)
- [Modeling Long- and Short-Term Temporal Patterns with Deep Neural Networks](https://mlsanity.com/paper/1703.07015/) (2017)
- [Deep learning for time series classification: a review](https://mlsanity.com/paper/1809.04356/) (2018)
- [Connecting the Dots: Multivariate Time Series Forecasting with Graph Neural Networks](https://mlsanity.com/paper/2005.11650/) (2020)
- [Recurrent Neural Networks for Time Series Forecasting: Current Status and Future Directions](https://mlsanity.com/paper/1909.00590/) (2019)
- [ETSformer: Exponential Smoothing Transformers for Time-series Forecasting](https://mlsanity.com/paper/2202.01381/) (2022)
- [N-HiTS: Neural Hierarchical Interpolation for Time Series Forecasting](https://mlsanity.com/paper/2201.12886/) (2022)
- [Temporal Pattern Attention for Multivariate Time Series Forecasting](https://mlsanity.com/paper/1809.04206/) (2018)
- [AdaRNN: Adaptive Learning and Forecasting of Time Series](https://mlsanity.com/paper/2108.04443/) (2021)
- [catch22: CAnonical Time-series CHaracteristics](https://mlsanity.com/paper/1901.10200/) (2019)
- [Characterizing Concept Drift](https://mlsanity.com/paper/1511.03816/) (2015)
- [Learning in High Dimension Always Amounts to Extrapolation](https://mlsanity.com/paper/2110.09485/) (2021)
- [Kaggle forecasting competitions: An overlooked learning opportunity](https://mlsanity.com/paper/2009.07701/) (2020)
- [Evaluating time series forecasting models: An empirical study on performance estimation methods](https://mlsanity.com/paper/1905.11744/) (2019)
- [Forecasting: theory and practice](https://mlsanity.com/paper/2012.03854/) (2020)
- [Measuring Data Leakage in Machine-Learning Models with Fisher Information](https://mlsanity.com/paper/2102.11673/) (2021)
- [Historical Inertia: A Neglected but Powerful Baseline for Long Sequence Time-series Forecasting](https://mlsanity.com/paper/2103.16349/) (2021)
- [Preformer: Predictive Transformer with Multi-Scale Segment-wise Correlations for Long-Term Time Series Forecasting](https://mlsanity.com/paper/2202.11356/) (2022)
- [Ensembles of Localised Models for Time Series Forecasting](https://mlsanity.com/paper/2012.15059/) (2020)
- [A better measure of relative prediction accuracy for model selection and model estimation](https://mlsanity.com/paper/2105.05249/) (2021)
- [Recurrent Neural Networks for Time Series Forecasting](https://mlsanity.com/paper/1901.00069/) (2019)
- [Neural Decomposition of Time-Series Data for Effective Generalization](https://mlsanity.com/paper/1705.09137/) (2017)
- [State-of-the-art predictive and prescriptive analytics for IEEE CIS 3rd Technical Challenge](https://mlsanity.com/paper/2112.03595/) (2021)
- [A Tutorial on Statistically Sound Pattern Discovery](https://mlsanity.com/paper/1709.03904/) (2017)
- [Error-feedback stochastic modeling strategy for time series forecasting with convolutional neural networks](https://mlsanity.com/paper/2002.00717/) (2020)

## Built on by (most cited first, 30 of 55)

- [Large Language Models Are Zero-Shot Time Series Forecasters](https://mlsanity.com/paper/2310.07820/) (2023)
- [fev-bench: A Realistic Benchmark for Time Series Forecasting](https://mlsanity.com/paper/2509.26468/) (2025)
- [It's TIME: Towards the Next Generation of Time Series Forecasting Benchmarks](https://mlsanity.com/paper/2602.12147/) (2026)
- [Pushing the Limits of Pre-training for Time Series Forecasting in the CloudOps Domain](https://mlsanity.com/paper/2310.05063/) (2023)
- [Ada-MSHyper: Adaptive Multi-Scale Hypergraph Transformer for Time Series Forecasting](https://mlsanity.com/paper/2410.23992/) (2024)
- [True Zero-Shot Inference of Dynamical Systems Preserving Long-Term Statistics](https://mlsanity.com/paper/2505.13192/) (2025)
- [Rethinking Evaluation in the Era of Time Series Foundation Models: (Un)known Information Leakage Challenges](https://mlsanity.com/paper/2510.13654/) (2025)
- [Neural Context Flows for Meta-Learning of Dynamical Systems](https://mlsanity.com/paper/2405.02154/) (2024)
- [There are no Champions in Supervised Long-Term Time Series Forecasting](https://mlsanity.com/paper/2502.14045/) (2025)
- [LETS Forecast: Learning Embedology for Time Series Forecasting](https://mlsanity.com/paper/2506.06454/) (2025)
- [Symbolic Regression as Feature Engineering Method for Machine and Deep Learning Regression Tasks](https://mlsanity.com/paper/2311.06028/) (2023)
- [Adaptive parameters identification for nonlinear dynamics using deep permutation invariant networks](https://mlsanity.com/paper/2501.11350/) (2025)
- [An Evaluation of Standard Statistical Models and LLMs on Time Series Forecasting](https://mlsanity.com/paper/2408.04867/) (2024)
- [Position: A Dynamical Systems Perspective is Needed to Advance Time Series Modeling](https://mlsanity.com/paper/2602.16864/) (2026)
- [Predict+Optimize Problem in Renewable Energy Scheduling](https://mlsanity.com/paper/2212.10723/) (2022)
- [The impact of data set similarity and diversity on transfer learning success in time series forecasting](https://mlsanity.com/paper/2404.06198/) (2024)
- [Scalable Transformer for High Dimensional Multivariate Time Series Forecasting](https://mlsanity.com/paper/2408.04245/) (2024)
- [On the retraining frequency of global models in retail demand forecasting](https://mlsanity.com/paper/2505.00356/) (2025)
- [The Pitfalls of Benchmarking in Algorithm Selection: What We Are Getting Wrong](https://mlsanity.com/paper/2505.07750/) (2025)
- [Time Series Foundation Models as Strong Baselines in Transportation Forecasting: A Large-Scale Benchmark Analysis](https://mlsanity.com/paper/2602.24238/) (2026)
- [Seeking SOTA: Time-Series Forecasting Must Adopt Taxonomy-Specific Evaluation to Dispel Illusory Gains](https://mlsanity.com/paper/2603.15506/) (2026)
- [Hierarchical learning, forecasting coherent spatio-temporal individual and aggregated building loads](https://mlsanity.com/paper/2301.12967/) (2023)
- [ModelRadar: Aspect-based Forecast Evaluation](https://mlsanity.com/paper/2504.00059/) (2025)
- [Navigating Pitfalls: Evaluating LLMs in Machine Learning Programming Education](https://mlsanity.com/paper/2505.18220/) (2025)
- [Multi-scale hypergraph meets LLMs: Aligning large language models for time series analysis](https://mlsanity.com/paper/2602.04369/) (2026)
- [Forecasting with Deep Learning: Beyond Average of Average of Average Performance](https://mlsanity.com/paper/2406.16590/) (2024)
- [Hybrid Adaptive Modeling in Process Monitoring: Leveraging Sequence Encoders and Physics-Informed Neural Networks](https://mlsanity.com/paper/2505.14252/) (2025)
- [The cost of ensembling: is it always worth combining?](https://mlsanity.com/paper/2506.04677/) (2025)
- [Intermittent time series forecasting: local vs global models](https://mlsanity.com/paper/2601.14031/) (2026)
- [Energy-Arena: A Dynamic Benchmark for Operational Energy Forecasting](https://mlsanity.com/paper/2604.24705/) (2026)

## Abstract (arXiv)

Machine Learning (ML) and Deep Learning (DL) methods are increasingly replacing traditional methods in many domains involved with important decision making activities. DL techniques tailor-made for specific tasks such as image recognition, signal processing, or speech analysis are being introduced at a fast pace with many improvements. However, for the domain of forecasting, the current state in the ML community is perhaps where other domains such as Natural Language Processing and Computer Vision were at several years ago. The field of forecasting has mainly been fostered by statisticians/econometricians; consequently the related concepts are not the mainstream knowledge among general ML practitioners. The different non-stationarities associated with time series challenge the data-driven ML models. Nevertheless, recent trends in the domain have shown that with the availability of massive amounts of time series, ML techniques are quite competent in forecasting, when related pitfalls are properly handled. Therefore, in this work we provide a tutorial-like compilation of the details of one of the most important steps in the overall forecasting process, namely the evaluation. This way, we intend to impart the information of forecast evaluation to fit the context of ML, as means of bridging the knowledge gap between traditional methods of forecasting and state-of-the-art ML techniques. We elaborate on the different problematic characteristics of time series such as non-normalities and non-stationarities and how they are associated with common pitfalls in forecast evaluation. Best practices in forecast evaluation are outlined with respect to the different steps such as data partitioning, error calculation, statistical testing, and others. Further guidelines are also provided along selecting valid and suitable error measures depending on the specific characteristics of the dataset at hand.