# Loss-Guided Pretraining Data Selection for Time-Series Foundation Models

arXiv: https://arxiv.org/abs/2609.37255 · HTML: https://mlsanity.com/paper/2609.37255/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 22 of its 27 references that are in the map.
- First posted on arXiv 29 Sep 2026.

## Builds on (most cited first, 22 of 22)

- [Autoformer: Decomposition Transformers with Auto-Correlation for Long-Term Series Forecasting](https://mlsanity.com/paper/2106.13008/) (2021)
- [Chronos: Learning the Language of Time Series](https://mlsanity.com/paper/2403.07815/) (2024)
- [Deduplicating Training Data Makes Language Models Better](https://mlsanity.com/paper/2107.06499/) (2021)
- [A decoder-only foundation model for time-series forecasting](https://mlsanity.com/paper/2310.10688/) (2023)
- [Unified Training of Universal Time Series Forecasting Transformers](https://mlsanity.com/paper/2402.02592/) (2024)
- [Estimating Training Data Influence by Tracing Gradient Descent](https://mlsanity.com/paper/2002.08484/) (2020)
- [Deep Learning on a Data Diet: Finding Important Examples Early in Training](https://mlsanity.com/paper/2107.07075/) (2021)
- [LESS: Selecting Influential Data for Targeted Instruction Tuning](https://mlsanity.com/paper/2402.04333/) (2024)
- [DoReMi: Optimizing Data Mixtures Speeds Up Language Model Pretraining](https://mlsanity.com/paper/2305.10429/) (2023)
- [Not All Samples Are Created Equal: Deep Learning with Importance Sampling](https://mlsanity.com/paper/1803.00942/) (2018)
- [SemDeDup: Data-efficient learning at web-scale through semantic deduplication](https://mlsanity.com/paper/2303.09540/) (2023)
- [Time-MoE: Billion-Scale Time Series Foundation Models with Mixture of Experts](https://mlsanity.com/paper/2409.16040/) (2024)
- [A Survey on Data Selection for Language Models](https://mlsanity.com/paper/2402.16827/) (2024)
- [GIFT-Eval: A Benchmark For General Time Series Forecasting Model Evaluation](https://mlsanity.com/paper/2410.10393/) (2024)
- [Data Mixing Laws: Optimizing Data Mixtures by Predicting Language Modeling Performance](https://mlsanity.com/paper/2403.16952/) (2024)
- [Sundial: A Family of Highly Capable Time Series Foundation Models](https://mlsanity.com/paper/2502.00816/) (2025)
- [DoGE: Domain Reweighting with Generalization Estimation](https://mlsanity.com/paper/2310.15393/) (2023)
- [Timer-XL: Long-Context Transformers for Unified Time Series Forecasting](https://mlsanity.com/paper/2410.04803/) (2024)
- [Efficient Online Data Mixing For Language Model Pre-Training](https://mlsanity.com/paper/2312.02406/) (2023)
- [Adaptive Data Optimization: Dynamic Sample Selection with Scaling Laws](https://mlsanity.com/paper/2410.11820/) (2024)
- [TimeInf: Time Series Data Contribution via Influence Functions](https://mlsanity.com/paper/2407.15247/) (2024)
- [Rating Quality of Diverse Time Series Data by Meta-learning from LLM Judgment](https://mlsanity.com/paper/2506.01290/) (2025)

## Abstract (arXiv)

Time series foundation models (TSFMs) are pretrained on heterogeneous collections containing billions of observations, yet their training windows are typically sampled without estimating whether they provide useful learning signal. We introduce a static data-selection framework that scores each window with a reference forecaster and retains an intermediate interval within every source dataset. Specifically, we connect forecasting loss to optimization difficulty by showing that normalized squared loss controls the per-sample gradient norm under a local Jacobian condition. We then define a reference loss score and apply dataset-stratified selection to preserve the diversity of samples. Across various TSFM architectures, our method outperforms random selection by an absolute margin and even improves both relative MASE and CRPS over full-data pretraining by retaining fewer candidate pretraining windows. Further analyses show strong cross-scale and cross-architecture score correlations, indicating that a small reference model can often select data for larger targets, provided that the reference and target share compatible difficulty orderings.