# Distillation of Synthetic Data for Time Series Foundation Models

arXiv: https://arxiv.org/abs/2609.09586 · HTML: https://mlsanity.com/paper/2609.09586/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Placed in Model collapse from synthetic data, one of 13 directions in Synthetic data and self-improvement for LLMs.
- Builds on 12 of its 16 references that are in the map.
- First posted on arXiv 9 Sep 2026.

## Placed in

- Synthetic data and self-improvement for LLMs / [Model collapse from synthetic data](https://mlsanity.com/synth/model-collapse-from-synthetic-data/)

## Builds on (most cited first, 12 of 12)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [A Time Series is Worth 64 Words: Long-term Forecasting with Transformers](https://mlsanity.com/paper/2211.14730/) (2022)
- [Chronos: Learning the Language of Time Series](https://mlsanity.com/paper/2403.07815/) (2024)
- [A decoder-only foundation model for time-series forecasting](https://mlsanity.com/paper/2310.10688/) (2023)
- [TabPFN: A Transformer That Solves Small Tabular Classification Problems in a Second](https://mlsanity.com/paper/2207.01848/) (2022)
- [Transformers Can Do Bayesian Inference](https://mlsanity.com/paper/2112.10510/) (2021)
- [Chronos-2: From Univariate to Universal Forecasting](https://mlsanity.com/paper/2510.15821/) (2025)
- [ForecastPFN: Synthetically-Trained Zero-Shot Forecasting](https://mlsanity.com/paper/2311.01933/) (2023)
- [Moirai 2.0: When Less Is More for Time Series Forecasting](https://mlsanity.com/paper/2511.11698/) (2025)
- [Toto 2.0: Time Series Forecasting Enters the Scaling Era](https://mlsanity.com/paper/2605.20119/) (2026)
- [TempoPFN: Synthetic Pre-training of Linear RNNs for Zero-shot Time Series Forecasting](https://mlsanity.com/paper/2510.25502/) (2025)
- [TiRex-2: Generalizing TiRex to Multivariate Data and Streaming](https://mlsanity.com/paper/2607.01204/) (2026)

## Abstract (arXiv)

Time series foundation models (TSFMs) are increasingly pre-trained on synthetically generated time series trajectories, where the data generating process is known. Current pre-training recipes are based on loss objectives which compare TSFM outputs to realized future values of each trajectory. We instead propose loss objectives which compare TSFM outputs to the conditional forecast distribution of each trajectory, a procedure we call synthetic data distillation (SDD). SDD corresponds to a Rao-Blackwellization of the training objective, in that it leaves the expectation of stochastic gradients unchanged while provably reducing the covariance of the stochastic gradient under the Loewner partial ordering. We empirically validate SDD on a TSFM model family of sizes from 4M to 2.5B parameters, and observe faster convergence of validation loss at every model size: on Gaussian Process data, SDD attains or improves upon the Status Quo loss whilst requiring 10%-40% less training iterations.