# Learning Spatio-Temporal Foundation Models from Pure Synthetic Data

arXiv: https://arxiv.org/abs/2607.16251 · HTML: https://mlsanity.com/paper/2607.16251/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Placed in Model collapse from synthetic data, one of 13 directions in Synthetic data and self-improvement for LLMs.
- Builds on 52 of its 62 references that are in the map.
- First posted on arXiv 27 Jun 2026.

## Placed in

- Synthetic data and self-improvement for LLMs / [Model collapse from synthetic data](https://mlsanity.com/synth/model-collapse-from-synthetic-data/)

## Builds on (most cited first, 30 of 52)

- [RoFormer: Enhanced Transformer with Rotary Position Embedding](https://mlsanity.com/paper/2104.09864/) (2021)
- [Self-Supervised Learning from Images with a Joint-Embedding Predictive Architecture](https://mlsanity.com/paper/2301.08243/) (2023)
- [Chronos: Learning the Language of Time Series](https://mlsanity.com/paper/2403.07815/) (2024)
- [A decoder-only foundation model for time-series forecasting](https://mlsanity.com/paper/2310.10688/) (2023)
- [Unified Training of Universal Time Series Forecasting Transformers](https://mlsanity.com/paper/2402.02592/) (2024)
- [MOMENT: A Family of Open Time-series Foundation Models](https://mlsanity.com/paper/2402.03885/) (2024)
- [Time Series Forecasting With Deep Learning: A Survey](https://mlsanity.com/paper/2004.13408/) (2020)
- [ClimaX: A foundation model for weather and climate](https://mlsanity.com/paper/2301.10343/) (2023)
- [Time-MoE: Billion-Scale Time Series Foundation Models with Mixture of Experts](https://mlsanity.com/paper/2409.16040/) (2024)
- [Gated Attention for Large Language Models: Non-linearity, Sparsity, and Attention-Sink-Free](https://mlsanity.com/paper/2505.06708/) (2025)
- [Structured Sequence Modeling with Graph Convolutional Recurrent Networks](https://mlsanity.com/paper/1612.07659/) (2016)
- [Soft-DTW: a Differentiable Loss Function for Time-Series](https://mlsanity.com/paper/1703.01541/) (2017)
- [LeJEPA: Provable and Scalable Self-Supervised Learning Without the Heuristics](https://mlsanity.com/paper/2511.08544/) (2025)
- [Foundation Models for Time Series Analysis: A Tutorial and Survey](https://mlsanity.com/paper/2403.14735/) (2024)
- [UniTime: A Language-Empowered Unified Model for Cross-Domain Time Series Forecasting](https://mlsanity.com/paper/2310.09751/) (2023)
- [Spatio-Temporal Graph Neural Networks for Predictive Learning in Urban Computing: A Survey](https://mlsanity.com/paper/2303.14483/) (2023)
- [LargeST: A Benchmark Dataset for Large-Scale Traffic Forecasting](https://mlsanity.com/paper/2306.08259/) (2023)
- [Deep Learning for Time Series Forecasting: Tutorial and Literature Survey](https://mlsanity.com/paper/2004.10240/) (2020)
- [Large Models for Time Series and Spatio-Temporal Data: A Survey and Outlook](https://mlsanity.com/paper/2310.10196/) (2023)
- [Self-Supervised Learning for Time Series Analysis: Taxonomy, Progress, and Prospects](https://mlsanity.com/paper/2306.10125/) (2023)
- [Moirai-MoE: Empowering Time Series Foundation Models with Sparse Mixture of Experts](https://mlsanity.com/paper/2410.10469/) (2024)
- [UniST: A Prompt-Empowered Universal Model for Urban Spatio-Temporal Prediction](https://mlsanity.com/paper/2402.11838/) (2024)
- [Hyper-Connections](https://mlsanity.com/paper/2409.19606/) (2024)
- [DiffSTG: Probabilistic Spatio-Temporal Graph Forecasting with Denoising Diffusion Models](https://mlsanity.com/paper/2301.13629/) (2023)
- [UrbanGPT: Spatio-Temporal Large Language Models](https://mlsanity.com/paper/2403.00813/) (2024)
- [AirFormer: Predicting Nationwide Air Quality in China with Transformers](https://mlsanity.com/paper/2211.15979/) (2022)
- [Shape and Time Distortion Loss for Training Deep Time Series Forecasting Models](https://mlsanity.com/paper/1909.09020/) (2019)
- [Moirai 2.0: When Less Is More for Time Series Forecasting](https://mlsanity.com/paper/2511.11698/) (2025)
- [A Survey on Diffusion Models for Time Series and Spatio-Temporal Data](https://mlsanity.com/paper/2404.18886/) (2024)
- [DiffTraj: Generating GPS Trajectory with Diffusion Probabilistic Model](https://mlsanity.com/paper/2304.11582/) (2023)

## Abstract (arXiv)

Spatio-Temporal Foundation Models (STFMs) aim to learn generalizable representations of complex dynamical systems across space and time. However, existing approaches suffer from distributional bias in real-world pre-training data, structural bottlenecks of autoregressive or diffusion-based paradigms, and objectives that overemphasize point-wise reconstruction in noisy observation space.We propose NeoST, the first spatio-temporal foundation model pre-trained solely on procedurally generated synthetic systems. NeoST introduces a scalable synthetic pre-training corpus to mitigate real-world bias, a latent-space reasoning architecture that generates and iteratively refines multiple future trajectories without sequential error accumulation, and latent-space objectives that emphasize structural dynamics and enable inference-time correction under distribution shifts.Extensive experiments across diverse real-world benchmarks show that NeoST consistently outperforms existing STFMs in diverse real-world spatio-temporal systems, achieves superior long-horizon stability and inference efficiency.