# fev-bench: A Realistic Benchmark for Time Series Forecasting

arXiv: https://arxiv.org/abs/2509.26468 · HTML: https://mlsanity.com/paper/2509.26468/

## Key facts

- Cited by 64 papers in the map (data through 14 Oct 2026).
- Builds on 33 of its 84 references that are in the map.
- First posted on arXiv 30 Sep 2025.

## Builds on (most cited first, 30 of 33)

- [A Time Series is Worth 64 Words: Long-term Forecasting with Transformers](https://mlsanity.com/paper/2211.14730/) (2022)
- [Autoformer: Decomposition Transformers with Auto-Correlation for Long-Term Series Forecasting](https://mlsanity.com/paper/2106.13008/) (2021)
- [Are Transformers Effective for Time Series Forecasting?](https://mlsanity.com/paper/2205.13504/) (2022)
- [Chatbot Arena: An Open Platform for Evaluating LLMs by Human Preference](https://mlsanity.com/paper/2403.04132/) (2024)
- [DeepAR: Probabilistic Forecasting with Autoregressive Recurrent Networks](https://mlsanity.com/paper/1704.04110/) (2017)
- [Modeling Long- and Short-Term Temporal Patterns with Deep Neural Networks](https://mlsanity.com/paper/1703.07015/) (2017)
- [Temporal Fusion Transformers for Interpretable Multi-horizon Time Series Forecasting](https://mlsanity.com/paper/1912.09363/) (2019)
- [Chronos: Learning the Language of Time Series](https://mlsanity.com/paper/2403.07815/) (2024)
- [A decoder-only foundation model for time-series forecasting](https://mlsanity.com/paper/2310.10688/) (2023)
- [Unified Training of Universal Time Series Forecasting Transformers](https://mlsanity.com/paper/2402.02592/) (2024)
- [Datasets: A Community Library for Natural Language Processing](https://mlsanity.com/paper/2109.02846/) (2021)
- [Monash Time Series Forecasting Archive](https://mlsanity.com/paper/2105.06643/) (2021)
- [TFB: Towards Comprehensive and Fair Benchmarking of Time Series Forecasting Methods](https://mlsanity.com/paper/2403.20150/) (2024)
- [GIFT-Eval: A Benchmark For General Time Series Forecasting Model Evaluation](https://mlsanity.com/paper/2410.10393/) (2024)
- [Sundial: A Family of Highly Capable Time Series Foundation Models](https://mlsanity.com/paper/2502.00816/) (2025)
- [TabArena: A Living Benchmark for Machine Learning on Tabular Data](https://mlsanity.com/paper/2506.16791/) (2025)
- [TiRex: Zero-Shot Forecasting Across Long and Short Horizons with Enhanced In-Context Learning](https://mlsanity.com/paper/2505.23719/) (2025)
- [Exploring Progress in Multivariate Time Series Forecasting: Comprehensive Benchmarking and Heterogeneity Analysis](https://mlsanity.com/paper/2310.06119/) (2023)
- [Forecasting day-ahead electricity prices: A review of state-of-the-art algorithms, best practices and an open-access benchmark](https://mlsanity.com/paper/2008.08004/) (2020)
- [sktime: A Unified Interface for Machine Learning with Time Series](https://mlsanity.com/paper/1909.07872/) (2019)
- [Darts: User-Friendly Modern Machine Learning for Time Series](https://mlsanity.com/paper/2110.03224/) (2021)
- [TSFM-Bench: A Comprehensive and Unified Benchmark of Foundation Models for Time Series Forecasting](https://mlsanity.com/paper/2410.11802/) (2024)
- [Kaggle forecasting competitions: An overlooked learning opportunity](https://mlsanity.com/paper/2009.07701/) (2020)
- [Forecast Evaluation for Data Scientists: Common Pitfalls and Best Practices](https://mlsanity.com/paper/2203.10716/) (2022)
- [AutoGluon-TimeSeries: AutoML for Probabilistic Time Series Forecasting](https://mlsanity.com/paper/2308.05566/) (2023)
- [ProbTS: Benchmarking Point and Distributional Forecasting across Diverse Prediction Horizons](https://mlsanity.com/paper/2310.07446/) (2023)
- [A machine learning approach for forecasting hierarchical time series](https://mlsanity.com/paper/2006.00630/) (2020)
- [ChronosX: Adapting Pretrained Time Series Models with Exogenous Variables](https://mlsanity.com/paper/2503.12107/) (2025)
- [LibCity: A Unified Library Towards Efficient and Comprehensive Urban Spatial-Temporal Prediction](https://mlsanity.com/paper/2304.14343/) (2023)
- [AutoMixer for Improved Multivariate Time-Series Forecasting on Business and IT Observability Data](https://mlsanity.com/paper/2310.20280/) (2023)

## Built on by (most cited first, 30 of 64)

- [Chronos-2: From Univariate to Universal Forecasting](https://mlsanity.com/paper/2510.15821/) (2025)
- [Moirai 2.0: When Less Is More for Time Series Forecasting](https://mlsanity.com/paper/2511.11698/) (2025)
- [It's TIME: Towards the Next Generation of Time Series Forecasting Benchmarks](https://mlsanity.com/paper/2602.12147/) (2026)
- [Beyond IID: How General Are Tabular Foundation Models, Really?](https://mlsanity.com/paper/2606.30410/) (2026)
- [TempoPFN: Synthetic Pre-training of Linear RNNs for Zero-shot Time Series Forecasting](https://mlsanity.com/paper/2510.25502/) (2025)
- [TabPFN-3: Technical Report](https://mlsanity.com/paper/2605.13986/) (2026)
- [Rethinking Evaluation in the Era of Time Series Foundation Models: (Un)known Information Leakage Challenges](https://mlsanity.com/paper/2510.13654/) (2025)
- [TiRex-2: Generalizing TiRex to Multivariate Data and Streaming](https://mlsanity.com/paper/2607.01204/) (2026)
- [Falcon-X: A Time Series Foundation Model for Heterogeneous Multivariate Modeling](https://mlsanity.com/paper/2605.27286/) (2026)
- [Learning the Context of Errors: Black-Box Online Adaptation of Time Series Foundation Models](https://mlsanity.com/paper/2606.14222/) (2026)
- [Position: Universal Time Series Foundation Models Rest on a Category Error](https://mlsanity.com/paper/2602.05287/) (2026)
- [Impermanent: A Live Benchmark for Temporal Generalization in Time Series Forecasting](https://mlsanity.com/paper/2603.08707/) (2026)
- [Baguan-TS: A Sequence-Native In-Context Learning Model for Time Series Forecasting with Covariates](https://mlsanity.com/paper/2603.17439/) (2026)
- [TS-ICL: A Flexible Time-Indexed Foundation Model for Time Series via In-Context Learning](https://mlsanity.com/paper/2606.05878/) (2026)
- [Post-Training in Time Series Foundation Models: A Unifying Framework](https://mlsanity.com/paper/2607.20002/) (2026)
- [Into the ORBIT for Time Series: Training Regimes for Foundation Models](https://mlsanity.com/paper/2608.13262/) (2026)
- [Time-Aware Prior Fitted Networks for Zero-Shot Forecasting with Exogenous Variables](https://mlsanity.com/paper/2603.15802/) (2026)
- [A Foundation Model for Instruction-Conditioned In-Context Time Series Tasks](https://mlsanity.com/paper/2603.22586/) (2026)
- [MulTaBench: Benchmarking Multimodal Tabular Learning with Text and Image](https://mlsanity.com/paper/2605.10616/) (2026)
- [Investigating simple target-covariate relationships for Chronos-2 and TabPFN-TS](https://mlsanity.com/paper/2605.12200/) (2026)
- [CITRAS-FM: Tiny Time Series Foundation Model for Covariate-Informed Zero-Shot Forecasting](https://mlsanity.com/paper/2606.10798/) (2026)
- [OpenMHC: Accelerating the Science of Wearable Foundation Models](https://mlsanity.com/paper/2607.16235/) (2026)
- [WPBench: A Comprehensive Benchmark for Wind Power Forecasting](https://mlsanity.com/paper/2609.24444/) (2026)
- [t0: A Time-Series Foundation Model for Forecasting with Context](https://mlsanity.com/paper/2609.24559/) (2026)
- [Time Series Foundation Models for Process Model Forecasting](https://mlsanity.com/paper/2512.07624/) (2025)
- [DiTS: Multimodal Diffusion Transformers Are Time Series Forecasters](https://mlsanity.com/paper/2602.06597/) (2026)
- [EIDOS: Latent-Space Predictive Learning for Time Series Foundation Models](https://mlsanity.com/paper/2602.14024/) (2026)
- [Energy-Arena: A Dynamic Benchmark for Operational Energy Forecasting](https://mlsanity.com/paper/2604.24705/) (2026)
- [Heterogeneous Scientific Foundation Model Collaboration](https://mlsanity.com/paper/2604.27351/) (2026)
- [AME-TS: Anchored Mixture-of-Experts for Time Series Forecasting](https://mlsanity.com/paper/2605.25166/) (2026)

## Abstract (arXiv)

Benchmark quality is critical for meaningful evaluation and sustained progress in time series forecasting, particularly with the rise of pretrained models. Existing benchmarks often have limited domain coverage or overlook real-world settings such as tasks with covariates. Their aggregation procedures frequently lack statistical rigor, making it unclear whether observed performance differences reflect true improvements or random variation. Many benchmarks lack consistent evaluation infrastructure or are too rigid for integration into existing pipelines. To address these gaps, we propose fev-bench, a benchmark of 100 forecasting tasks across seven domains, including 46 with covariates. Supporting the benchmark, we introduce fev, a lightweight Python library for forecasting evaluation emphasizing reproducibility and integration with existing workflows. Using fev, fev-bench employs principled aggregation with bootstrapped confidence intervals to report performance along two dimensions: win rates and skill scores. We report results on fev-bench for pretrained, statistical, and baseline models and identify promising future research directions.