# ModelRadar: Aspect-based Forecast Evaluation

arXiv: https://arxiv.org/abs/2504.00059 · HTML: https://mlsanity.com/paper/2504.00059/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 19 of its 61 references that are in the map.
- First posted on arXiv 31 Mar 2025.

## Builds on (most cited first, 19 of 19)

- [Informer: Beyond Efficient Transformer for Long Sequence Time-Series Forecasting](https://mlsanity.com/paper/2012.07436/) (2020)
- [A Time Series is Worth 64 Words: Long-term Forecasting with Transformers](https://mlsanity.com/paper/2211.14730/) (2022)
- [An Empirical Evaluation of Generic Convolutional and Recurrent Networks for Sequence Modeling](https://mlsanity.com/paper/1803.01271/) (2018)
- [Are Transformers Effective for Time Series Forecasting?](https://mlsanity.com/paper/2205.13504/) (2022)
- [DeepAR: Probabilistic Forecasting with Autoregressive Recurrent Networks](https://mlsanity.com/paper/1704.04110/) (2017)
- [Temporal Fusion Transformers for Interpretable Multi-horizon Time Series Forecasting](https://mlsanity.com/paper/1912.09363/) (2019)
- [N-BEATS: Neural basis expansion analysis for interpretable time series forecasting](https://mlsanity.com/paper/1905.10437/) (2019)
- [KAN: Kolmogorov-Arnold Networks](https://mlsanity.com/paper/2404.19756/) (2024)
- [Dilated Recurrent Neural Networks](https://mlsanity.com/paper/1710.02224/) (2017)
- [Forecast Evaluation for Data Scientists: Common Pitfalls and Best Practices](https://mlsanity.com/paper/2203.10716/) (2022)
- [SegRNN: Segment Recurrent Neural Network for Long-Term Time Series Forecasting](https://mlsanity.com/paper/2308.11200/) (2023)
- [Criteria for Classifying Forecasting Methods](https://mlsanity.com/paper/2212.03523/) (2022)
- [AR-Net: A simple Auto-Regressive Neural Network for time-series](https://mlsanity.com/paper/1911.12436/) (2019)
- [Ensembles of Localised Models for Time Series Forecasting](https://mlsanity.com/paper/2012.15059/) (2020)
- [PatchMixer: A Patch-Mixing Architecture for Long-Term Time Series Forecasting](https://mlsanity.com/paper/2310.00655/) (2023)
- [Cherry-Picking in Time Series Forecasting: How to Select Datasets to Make Your Model Shine](https://mlsanity.com/paper/2412.14435/) (2024)
- [Deep Non-Parametric Time Series Forecaster](https://mlsanity.com/paper/2312.14657/) (2023)
- [Model Selection for Time Series Forecasting: Empirical Analysis of Different Estimators](https://mlsanity.com/paper/2104.00584/) (2021)
- [Forecasting with Deep Learning: Beyond Average of Average of Average Performance](https://mlsanity.com/paper/2406.16590/) (2024)

## Built on by (most cited first, 2 of 2)

- [Grasynda: Graph-based Synthetic Time Series Generation](https://mlsanity.com/paper/2601.19668/) (2026)
- [Benchmarking Time Series Generation Methods for Privacy-Preserving Forecasting](https://mlsanity.com/paper/2608.10891/) (2026)

## Abstract (arXiv)

Accurate evaluation of forecasting models is essential for ensuring reliable predictions. Current practices for evaluating and comparing forecasting models focus on summarising performance into a single score, using metrics such as SMAPE. While convenient, averaging performance over all samples dilutes relevant information about model behavior under varying conditions. This limitation is especially problematic for time series forecasting, where multiple layers of averaging--across time steps, horizons, and multiple time series in a dataset--can mask relevant performance variations. We address this limitation by proposing ModelRadar, a framework for evaluating univariate time series forecasting models across multiple aspects, such as stationarity, presence of anomalies, or forecasting horizons. We demonstrate the advantages of this framework by comparing 24 forecasting methods, including classical approaches and different machine learning algorithms. NHITS, a state-of-the-art neural network architecture, performs best overall but its superiority varies with forecasting conditions. For instance, concerning the forecasting horizon, we found that NHITS (and also other neural networks) only outperforms classical approaches for multi-step ahead forecasting. Another relevant insight is that classical approaches such as ETS or Theta are notably more robust in the presence of anomalies. These and other findings highlight the importance of aspect-based model evaluation for both practitioners and researchers. ModelRadar is available as a Python package.