mlsanityData to 7 Oct 2026

Distillation of Synthetic Data for Time Series Foundation Models

arXiv 2609.09586 ↗First posted 9 Sep 2026stat.ML cs.LGcited by 0

On the map: Synthetic data and self-improvement for LLMs → Model collapse from synthetic data

Abstract (arXiv)

Time series foundation models (TSFMs) are increasingly pre-trained on synthetically generated time series trajectories, where the data generating process is known. Current pre-training recipes are based on loss objectives which compare TSFM outputs to realized future values of each trajectory. We instead propose loss objectives which compare TSFM outputs to the conditional forecast distribution of each trajectory, a procedure we call synthetic data distillation (SDD). SDD corresponds to a Rao-Blackwellization of the training objective, in that it leaves the expectation of stochastic gradients unchanged while provably reducing the covariance of the stochastic gradient under the Loewner partial ordering. We empirically validate SDD on a TSFM model family of sizes from 4M to 2.5B parameters, and observe faster convergence of validation loss at every model size: on Gaussian Process data, SDD attains or improves upon the Status Quo loss whilst requiring 10%-40% less training iterations.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Placed in Model collapse from synthetic data, one of 13 directions in Synthetic data and self-improvement for LLMs. Builds on 12 of its 16 references that are in the map. First posted on arXiv 9 Sep 2026.

Builds on

12 of 16 references.

References · 12

  1. 2017Attention Is All You Needcited by 59,932
  2. 2022A Time Series is Worth 64 Words: Long-term Forecasting with Transformerscited by 1,981
  3. 2024Chronos: Learning the Language of Time Seriescited by 712
  4. 2023A decoder-only foundation model for time-series forecastingcited by 650
  5. 2022TabPFN: A Transformer That Solves Small Tabular Classification Problems in a Secondcited by 643
  6. 2021Transformers Can Do Bayesian Inferencecited by 338
  7. 2025Chronos-2: From Univariate to Universal Forecastingcited by 158
  8. 2023ForecastPFN: Synthetically-Trained Zero-Shot Forecastingcited by 102
  9. 2025Moirai 2.0: When Less Is More for Time Series Forecastingcited by 59
  10. 2026Toto 2.0: Time Series Forecasting Enters the Scaling Eracited by 21
  11. 2025TempoPFN: Synthetic Pre-training of Linear RNNs for Zero-shot Time Series Forecastingcited by 16
  12. 2026TiRex-2: Generalizing TiRex to Multivariate Data and Streamingcited by 10

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2025Empowering Time Series Analysis with Synthetic Data: A Survey and Outlook in the Era of Foundation Models
  2. 2026GRAM: Correcting Frozen Time-Series Foundation Models via Graph-Retrieved Amplitude Memory
  3. 2025CauKer: Classification Time Series Foundation Models Can Be Pretrained on Synthetic Data
  4. 2025TIMED: Adversarial and Autoregressive Refinement of Diffusion-Based Time Series Generation
  5. 2023Predict, Refine, Synthesize: Self-Guiding Diffusion Models for Probabilistic Time Series Forecasting
  6. 2025Sundial: A Family of Highly Capable Time Series Foundation Models
  7. 2026Foundations without Fundamentals: Zero-Shot Blind Spots in Time Series FMs
  8. 2025Evaluating Temporal Plasticity in Foundation Time Series Models for Incremental Fine-tuning