mlsanityData to 14 Oct 2026

A Later Test Set Is Not a New Domain: Pretraining Familiarity Survives a Contamination-Free Hold-Out

arXiv 2609.10357 ↗First posted 9 Sep 2026cs.LGcited by 0

Not on the map. Why?

Abstract (arXiv)

Time-series foundation models are evaluated almost exclusively on public archives that predate them, so a strong score cannot be separated from having seen the test set during pretraining. The obvious remedy is a hold-out that postdates the models. We build one: thirteen forecasters -- four classical, three trained per dataset, six pretrained -- on seven groups drawn from five domains, every observation published after the last model was released, and every dataset rebuildable without an API key. Under this protocol pretrained models win 5 of 7 groups, lose one to a Theta baseline, and on daily exchange rates are indistinguishable from a seasonal naive forecast, along with every other method tested. We then ask what separates the wins from the losses, and report a negative result: the two intrinsic properties one would reach for -- seasonal strength and spectral entropy, measured on the input window -- do not account for the pattern, and seasonal strength is if anything negatively associated with the advantage. What does track it is corpus familiarity. Our largest gain (28% lower MASE than the best classical method, on weekly Wikipedia pageviews) falls on Wikipedia pageviews, the domain TimesFM's authors describe as the bulk of its pretraining corpus, at the same granularities and differing only in time window. Within the pretrained family, where every model forecasts identical series so that series difficulty cancels, the TimesFM family outranks the Chronos family by -0.53 ranks on Wikipedia against -0.09 everywhere else (1,500 vs. 754 series, Mann-Whitney p < 1e-5). We conclude that a temporal hold-out removes memorisation of a window but not familiarity with a domain, that benchmarks therefore need domain hold-outs stated relative to disclosed corpora, and that the practitioner's question is less which model is better than whether their domain is one the model was raised on.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 29 of its 45 references that are in the map. First posted on arXiv 9 Sep 2026.

Builds on

29 of 45 references.

References · 29

  1. 2017On Calibration of Modern Neural Networkscited by 4,960
  2. 2020Informer: Beyond Efficient Transformer for Long Sequence Time-Series Forecastingcited by 2,169
  3. 2022A Time Series is Worth 64 Words: Long-term Forecasting with Transformerscited by 2,039
  4. 2021Autoformer: Decomposition Transformers with Auto-Correlation for Long-Term Series Forecastingcited by 1,676
  5. 2022Are Transformers Effective for Time Series Forecasting?cited by 1,424
  6. 2017DeepAR: Probabilistic Forecasting with Autoregressive Recurrent Networkscited by 1,013
  7. 2023iTransformer: Inverted Transformers Are Effective for Time Series Forecastingcited by 931
  8. 2019Temporal Fusion Transformers for Interpretable Multi-horizon Time Series Forecastingcited by 761
  9. 2024Chronos: Learning the Language of Time Seriescited by 759
  10. 2019N-BEATS: Neural basis expansion analysis for interpretable time series forecastingcited by 753
  11. 2023A decoder-only foundation model for time-series forecastingcited by 692
  12. 2024Unified Training of Universal Time Series Forecasting Transformerscited by 614
17 more
  1. 2024MOMENT: A Family of Open Time-series Foundation Modelscited by 482
  2. 2018Accurate Uncertainties for Deep Learning Using Calibrated Regressioncited by 422
  3. 2023NLP Evaluation in trouble: On the Need to Measure LLM Data Contamination for each Benchmarkcited by 348
  4. 2021Monash Time Series Forecasting Archivecited by 267
  5. 2024GIFT-Eval: A Benchmark For General Time Series Forecasting Model Evaluationcited by 212
  6. 2024Timer: Generative Pre-trained Transformers Are Large Time Series Modelscited by 208
  7. 2025Sundial: A Family of Highly Capable Time Series Foundation Modelscited by 180
  8. 2024Are Language Models Actually Useful for Time Series Forecasting?cited by 168
  9. 2023TimeGPT-1cited by 158
  10. 2024Benchmark Data Contamination of Large Language Models: A Surveycited by 147
  11. 2024Tiny Time Mixers (TTMs): Fast Pre-trained Models for Enhanced Zero/Few-Shot Forecasting of Multivariate Time Seriescited by 127
  12. 2023Lag-Llama: Towards Foundation Models for Probabilistic Time Series Forecastingcited by 113
  13. 2025fev-bench: A Realistic Benchmark for Time Series Forecastingcited by 64
  14. 2024Specialized Foundation Models Struggle to Beat Supervised Baselinescited by 24
  15. 2025Rethinking Evaluation in the Era of Time Series Foundation Models: (Un)known Information Leakage Challengescited by 16
  16. 2025Performance of Zero-Shot Time Series Foundation Models on Cloud Datacited by 10
  17. 2026TSFMAudit: Data Contamination Auditing in Forecasting Time Series Foundation Modelscited by 3

Built on by

Not cited yet.