# Do Time Series Foundation Model Benchmarks Hide Regime-Dependent Failures? Evidence from Traffic Speed Forecasting

arXiv: https://arxiv.org/abs/2606.18367 · HTML: https://mlsanity.com/paper/2606.18367/

## Key facts

- Cited by 5 papers in the map (data through 14 Oct 2026).
- Builds on 11 of its 17 references that are in the map.
- First posted on arXiv 16 Jun 2026.

## Builds on (most cited first, 11 of 11)

- [Spatio-Temporal Graph Convolutional Networks: A Deep Learning Framework for Traffic Forecasting](https://mlsanity.com/paper/1709.04875/) (2017)
- [Chronos: Learning the Language of Time Series](https://mlsanity.com/paper/2403.07815/) (2024)
- [Unified Training of Universal Time Series Forecasting Transformers](https://mlsanity.com/paper/2402.02592/) (2024)
- [Adaptive Conformal Inference Under Distribution Shift](https://mlsanity.com/paper/2106.00170/) (2021)
- [GIFT-Eval: A Benchmark For General Time Series Forecasting Model Evaluation](https://mlsanity.com/paper/2410.10393/) (2024)
- [TSFM-Bench: A Comprehensive and Unified Benchmark of Foundation Models for Time Series Forecasting](https://mlsanity.com/paper/2410.11802/) (2024)
- [Relational Conformal Prediction for Correlated Time Series](https://mlsanity.com/paper/2502.09443/) (2025)
- [Adaptive Modeling of Uncertainties for Traffic Forecasting](https://mlsanity.com/paper/2303.09273/) (2023)
- [Probabilistic Traffic Forecasting with Dynamic Regression](https://mlsanity.com/paper/2301.06650/) (2023)
- [Beyond Accuracy: Are Time Series Foundation Models Well-Calibrated?](https://mlsanity.com/paper/2510.16060/) (2025)
- [Adaptive Regime-Switching Forecasts with Distribution-Free Uncertainty: Deep Switching State-Space Models Meet Conformal Prediction](https://mlsanity.com/paper/2512.03298/) (2025)

## Built on by (most cited first, 5 of 5)

- [When Should Active RAG Retrieve? A Budget-Aware Evaluation of Utility, Calibration, and Cost](https://mlsanity.com/paper/2607.24010/) (2026)
- [Toward User-Conditioned Evaluation of Personal LLM Agents under Temporal Interventions](https://mlsanity.com/paper/2607.21635/) (2026)
- [Evidence-Unit Fairness and the Limits of Query-Adaptive Sparse-Dense Fusion in Financial Document Retrieval](https://mlsanity.com/paper/2608.00183/) (2026)
- [From Non-Convex Self-Concordant Regularization to Scalable Quasi-Newton Training of PINNs](https://mlsanity.com/paper/2608.04206/) (2026)
- [Noise Floor Audit for Agent Benchmarks](https://mlsanity.com/paper/2608.22331/) (2026)

## Abstract (arXiv)

Standard benchmarks evaluate time series foundation models (TSFMs) using aggregate metrics, but these can mask severe failures in critical operating regimes. We introduce regime-stratified evaluation and apply it to three TSFMs on two standard traffic speed benchmarks. Traffic exhibits abrupt regime switching between free-flow and congested states, producing bimodal speed distributions during transitions. When we stratify by traffic regime, both accuracy and prediction-interval coverage degrade sharply during transitions: transition-regime MAE reaches 11 mph (versus 3 mph overall), and empirical coverage of 90% prediction intervals drops as low as 55%. These failures are invisible in aggregate metrics because free-flow observations dominate the sample. A simple historical conditional baseline (sampling from per-sensor training distributions) achieves better transition coverage than any TSFM, but has far worse overall accuracy. We propose bimodal mixture augmentation (BMA), a post-hoc method that combines TSFM forecasts with historical distributional knowledge, approaching the historical baseline's transition coverage while preserving the TSFM's accuracy. Our results suggest that TSFM benchmarks should incorporate regime-aware evaluation to surface failures that aggregate metrics hide.