mlsanityData to 14 Oct 2026

Beyond Average Error through Oracle-Informed Stress Tests for Time-Series Forecasting

arXiv 2609.22820 ↗First posted 19 Sep 2026cs.LGcited by 0

Not on the map. Why?

Abstract (arXiv)

Average squared error cannot reveal whether forecasting performance degrades because the future becomes less predictable or because forecasts move farther from the conditional mean. We introduce paired, mechanism-controlled stress tests that decompose changes in expected squared error at each lead time into environmental risk and forecast-oracle distance, using an origin-conditioned predictive oracle unavailable to the evaluated models. Three end-to-end controls have known attribution. Specifically, the null, environmental-only, and information-gap controls verify that the pipeline assigns changes to the correct component. We then apply the benchmark to 24 deployable forecasters. Under frequent switching, 14 methods have higher realized MSE but lower oracle distance; under outlier-variance feedback, 19 have higher MSE but lower scale-standardized MSE. Short- and long-lead stress-response rankings have Spearman correlation 0.624, revealing substantial horizon-dependent reordering. We then study multivariate relation shifts. Across six models and three coupling severities, oracle distance accounts for only 0.7-3.9% of the decomposed expected-risk increase, and environmental-risk majority persists in an eight-channel system and a matched-difficulty audit of Ring, Block, and Hub relations. Finally, prespecified contrasts on independent data-generating process (DGP) realizations show that several visually compelling discovery profiles, including trend accumulation and the hypothesized switching reversal, do not replicate. The benchmark thus combines component-wise diagnosis with a held-out stability audit. It complements real-data out-of-distribution evaluation, which measures performance under realistic shifts when exact oracle attribution is unavailable.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 45 of its 54 references that are in the map. First posted on arXiv 19 Sep 2026.

Builds on

45 of 54 references.

References · 45

  1. 2020Informer: Beyond Efficient Transformer for Long Sequence Time-Series Forecastingcited by 2,169
  2. 2022A Time Series is Worth 64 Words: Long-term Forecasting with Transformerscited by 2,039
  3. 2021Autoformer: Decomposition Transformers with Auto-Correlation for Long-Term Series Forecastingcited by 1,676
  4. 2022Are Transformers Effective for Time Series Forecasting?cited by 1,424
  5. 2020WILDS: A Benchmark of in-the-Wild Distribution Shiftscited by 1,375
  6. 2022FEDformer: Frequency Enhanced Decomposed Transformer for Long-term Series Forecastingcited by 1,210
  7. 2022TimesNet: Temporal 2D-Variation Modeling for General Time Series Analysiscited by 1,058
  8. 2020In Search of Lost Domain Generalizationcited by 1,054
  9. 2024Chronos: Learning the Language of Time Seriescited by 759
  10. 2019N-BEATS: Neural basis expansion analysis for interpretable time series forecastingcited by 753
  11. 2023A decoder-only foundation model for time-series forecastingcited by 692
  12. 2024Unified Training of Universal Time Series Forecasting Transformerscited by 614
33 more
  1. 2023Large Language Models Are Zero-Shot Time Series Forecasterscited by 500
  2. 2024MOMENT: A Family of Open Time-series Foundation Modelscited by 482
  3. 2022Non-stationary Transformers: Exploring the Stationarity in Time Series Forecastingcited by 391
  4. 2024TimeMixer: Decomposable Multiscale Mixing for Time Series Forecastingcited by 346
  5. 2024Time-MoE: Billion-Scale Time Series Foundation Models with Mixture of Expertscited by 287
  6. 2021Monash Time Series Forecasting Archivecited by 267
  7. 2024TFB: Towards Comprehensive and Fair Benchmarking of Time Series Forecasting Methodscited by 249
  8. 2024TimeXer: Empowering Transformers for Time Series Forecasting with Exogenous Variablescited by 248
  9. 2023TSMixer: An All-MLP Architecture for Time Series Forecastingcited by 240
  10. 2024GIFT-Eval: A Benchmark For General Time Series Forecasting Model Evaluationcited by 212
  11. 2024Timer: Generative Pre-trained Transformers Are Large Time Series Modelscited by 208
  12. 2021Accounting for Variance in Machine Learning Benchmarkscited by 201
  13. 2025Chronos-2: From Univariate to Universal Forecastingcited by 194
  14. 2025Sundial: A Family of Highly Capable Time Series Foundation Modelscited by 180
  15. 2024Tiny Time Mixers (TTMs): Fast Pre-trained Models for Enhanced Zero/Few-Shot Forecasting of Multivariate Time Seriescited by 127
  16. 2023Koopa: Learning Non-stationary Time Series Dynamics with Koopman Predictorscited by 113
  17. 2024Moirai-MoE: Empowering Time Series Foundation Models with Sparse Mixture of Expertscited by 95
  18. 2022Wild-Time: A Benchmark of in-the-Wild Distribution Shift over Timecited by 90
  19. 2024VisionTS: Visual Masked Autoencoders Are Free-Lunch Zero-Shot Time Series Forecasterscited by 88
  20. 2023OneNet: Enhancing Time Series Forecasting Models under Concept Drift by Online Ensemblingcited by 74
  21. 2025fev-bench: A Realistic Benchmark for Time Series Forecastingcited by 64
  22. 2024Frequency Adaptive Normalization For Non-stationary Time Series Forecastingcited by 47
  23. 2024TimeBridge: Non-Stationarity Matters for Long-term Time Series Forecastingcited by 40
  24. 2023ProbTS: Benchmarking Point and Distributional Forecasting across Diverse Prediction Horizonscited by 36
  25. 2024In-Context Fine-Tuning for Time-Series Foundation Modelscited by 35
  26. 2025TimePFN: Effective Multivariate Time Series Forecasting with Synthetic Datacited by 29
  27. 2024In-context Time Series Predictorcited by 23
  28. 2025Pitfalls in Evaluating Language Model Forecasterscited by 23
  29. 2024Are Self-Attentions Effective for Time Series Forecasting?cited by 19
  30. 2025Lightweight Online Adaption for Time Series Foundation Model Forecastscited by 16
  31. 2024Benchmark Data Repositories for Better Benchmarkingcited by 10
  32. 2024The State of Data Curation at NeurIPS: An Assessment of Dataset Development Practices in the Datasets and Benchmarks Trackcited by 7
  33. 2026FinStressTS: A Parametric Synthetic Benchmark for Time-Series Forecasting in Financecited by 2

Built on by

Not cited yet.