A Later Test Set Is Not a New Domain: Pretraining Familiarity Survives a Contamination-Free Hold-Out
Not on the map. Why?
Abstract (arXiv)
Time-series foundation models are evaluated almost exclusively on public archives that predate them, so a strong score cannot be separated from having seen the test set during pretraining. The obvious remedy is a hold-out that postdates the models. We build one: thirteen forecasters -- four classical, three trained per dataset, six pretrained -- on seven groups drawn from five domains, every observation published after the last model was released, and every dataset rebuildable without an API key. Under this protocol pretrained models win 5 of 7 groups, lose one to a Theta baseline, and on daily exchange rates are indistinguishable from a seasonal naive forecast, along with every other method tested. We then ask what separates the wins from the losses, and report a negative result: the two intrinsic properties one would reach for -- seasonal strength and spectral entropy, measured on the input window -- do not account for the pattern, and seasonal strength is if anything negatively associated with the advantage. What does track it is corpus familiarity. Our largest gain (28% lower MASE than the best classical method, on weekly Wikipedia pageviews) falls on Wikipedia pageviews, the domain TimesFM's authors describe as the bulk of its pretraining corpus, at the same granularities and differing only in time window. Within the pretrained family, where every model forecasts identical series so that series difficulty cancels, the TimesFM family outranks the Chronos family by -0.53 ranks on Wikipedia against -0.09 everywhere else (1,500 vs. 754 series, Mann-Whitney p < 1e-5). We conclude that a temporal hold-out removes memorisation of a window but not familiarity with a domain, that benchmarks therefore need domain hold-outs stated relative to disclosed corpora, and that the practitioner's question is less which model is better than whether their domain is one the model was raised on.
Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 29 of its 45 references that are in the map. First posted on arXiv 9 Sep 2026.
Builds on
29 of 45 references.
References · 29
- 2017On Calibration of Modern Neural Networkscited by 4,960
- 2020Informer: Beyond Efficient Transformer for Long Sequence Time-Series Forecastingcited by 2,169
- 2022A Time Series is Worth 64 Words: Long-term Forecasting with Transformerscited by 2,039
- 2021Autoformer: Decomposition Transformers with Auto-Correlation for Long-Term Series Forecastingcited by 1,676
- 2022Are Transformers Effective for Time Series Forecasting?cited by 1,424
- 2017DeepAR: Probabilistic Forecasting with Autoregressive Recurrent Networkscited by 1,013
- 2023iTransformer: Inverted Transformers Are Effective for Time Series Forecastingcited by 931
- 2019Temporal Fusion Transformers for Interpretable Multi-horizon Time Series Forecastingcited by 761
- 2024Chronos: Learning the Language of Time Seriescited by 759
- 2019N-BEATS: Neural basis expansion analysis for interpretable time series forecastingcited by 753
- 2023A decoder-only foundation model for time-series forecastingcited by 692
- 2024Unified Training of Universal Time Series Forecasting Transformerscited by 614
17 more
- 2024MOMENT: A Family of Open Time-series Foundation Modelscited by 482
- 2018Accurate Uncertainties for Deep Learning Using Calibrated Regressioncited by 422
- 2023NLP Evaluation in trouble: On the Need to Measure LLM Data Contamination for each Benchmarkcited by 348
- 2021Monash Time Series Forecasting Archivecited by 267
- 2024GIFT-Eval: A Benchmark For General Time Series Forecasting Model Evaluationcited by 212
- 2024Timer: Generative Pre-trained Transformers Are Large Time Series Modelscited by 208
- 2025Sundial: A Family of Highly Capable Time Series Foundation Modelscited by 180
- 2024Are Language Models Actually Useful for Time Series Forecasting?cited by 168
- 2023TimeGPT-1cited by 158
- 2024Benchmark Data Contamination of Large Language Models: A Surveycited by 147
- 2024Tiny Time Mixers (TTMs): Fast Pre-trained Models for Enhanced Zero/Few-Shot Forecasting of Multivariate Time Seriescited by 127
- 2023Lag-Llama: Towards Foundation Models for Probabilistic Time Series Forecastingcited by 113
- 2025fev-bench: A Realistic Benchmark for Time Series Forecastingcited by 64
- 2024Specialized Foundation Models Struggle to Beat Supervised Baselinescited by 24
- 2025Rethinking Evaluation in the Era of Time Series Foundation Models: (Un)known Information Leakage Challengescited by 16
- 2025Performance of Zero-Shot Time Series Foundation Models on Cloud Datacited by 10
- 2026TSFMAudit: Data Contamination Auditing in Forecasting Time Series Foundation Modelscited by 3
Built on by
Not cited yet.