mlsanityData to 14 Oct 2026

Learning Spatio-Temporal Foundation Models from Pure Synthetic Data

arXiv 2607.16251 ↗First posted 27 Jun 2026cs.LGcited by 0

On the map: Synthetic data and self-improvement for LLMs → Model collapse from synthetic data

Abstract (arXiv)

Spatio-Temporal Foundation Models (STFMs) aim to learn generalizable representations of complex dynamical systems across space and time. However, existing approaches suffer from distributional bias in real-world pre-training data, structural bottlenecks of autoregressive or diffusion-based paradigms, and objectives that overemphasize point-wise reconstruction in noisy observation space.We propose NeoST, the first spatio-temporal foundation model pre-trained solely on procedurally generated synthetic systems. NeoST introduces a scalable synthetic pre-training corpus to mitigate real-world bias, a latent-space reasoning architecture that generates and iteratively refines multiple future trajectories without sequential error accumulation, and latent-space objectives that emphasize structural dynamics and enable inference-time correction under distribution shifts.Extensive experiments across diverse real-world benchmarks show that NeoST consistently outperforms existing STFMs in diverse real-world spatio-temporal systems, achieves superior long-horizon stability and inference efficiency.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in Model collapse from synthetic data, one of 13 directions in Synthetic data and self-improvement for LLMs. Builds on 52 of its 62 references that are in the map. First posted on arXiv 27 Jun 2026.

Builds on

52 of 62 references.

Closest ancestors · 1

Same direction, newest first

  1. 2025Empowering Time Series Analysis with Synthetic Data: A Survey and Outlook in the Era of Foundation Models14 Mar

Other references · 51

  1. 2021RoFormer: Enhanced Transformer with Rotary Position Embeddingcited by 4,703
  2. 2023Self-Supervised Learning from Images with a Joint-Embedding Predictive Architecturecited by 1,039
  3. 2024Chronos: Learning the Language of Time Seriescited by 759
  4. 2023A decoder-only foundation model for time-series forecastingcited by 692
  5. 2024Unified Training of Universal Time Series Forecasting Transformerscited by 614
  6. 2024MOMENT: A Family of Open Time-series Foundation Modelscited by 482
  7. 2020Time Series Forecasting With Deep Learning: A Surveycited by 459
  8. 2023ClimaX: A foundation model for weather and climatecited by 337
  9. 2024Time-MoE: Billion-Scale Time Series Foundation Models with Mixture of Expertscited by 287
  10. 2025Gated Attention for Large Language Models: Non-linearity, Sparsity, and Attention-Sink-Freecited by 283
  11. 2016Structured Sequence Modeling with Graph Convolutional Recurrent Networkscited by 275
  12. 2017Soft-DTW: a Differentiable Loss Function for Time-Seriescited by 271
38 more
  1. 2025LeJEPA: Provable and Scalable Self-Supervised Learning Without the Heuristicscited by 254
  2. 2024Foundation Models for Time Series Analysis: A Tutorial and Surveycited by 239
  3. 2023UniTime: A Language-Empowered Unified Model for Cross-Domain Time Series Forecastingcited by 144
  4. 2023Spatio-Temporal Graph Neural Networks for Predictive Learning in Urban Computing: A Surveycited by 140
  5. 2023LargeST: A Benchmark Dataset for Large-Scale Traffic Forecastingcited by 118
  6. 2020Deep Learning for Time Series Forecasting: Tutorial and Literature Surveycited by 117
  7. 2023Large Models for Time Series and Spatio-Temporal Data: A Survey and Outlookcited by 110
  8. 2023Self-Supervised Learning for Time Series Analysis: Taxonomy, Progress, and Prospectscited by 99
  9. 2024Moirai-MoE: Empowering Time Series Foundation Models with Sparse Mixture of Expertscited by 95
  10. 2024UniST: A Prompt-Empowered Universal Model for Urban Spatio-Temporal Predictioncited by 87
  11. 2024Hyper-Connectionscited by 84
  12. 2023DiffSTG: Probabilistic Spatio-Temporal Graph Forecasting with Denoising Diffusion Modelscited by 81
  13. 2024UrbanGPT: Spatio-Temporal Large Language Modelscited by 81
  14. 2022AirFormer: Predicting Nationwide Air Quality in China with Transformerscited by 73
  15. 2019Shape and Time Distortion Loss for Training Deep Time Series Forecasting Modelscited by 70
  16. 2025Moirai 2.0: When Less Is More for Time Series Forecastingcited by 69
  17. 2024A Survey on Diffusion Models for Time Series and Spatio-Temporal Datacited by 67
  18. 2023DiffTraj: Generating GPS Trajectory with Diffusion Probabilistic Modelcited by 65
  19. 2024ControlTraj: Controllable Trajectory Generation with Topology-Constrained Diffusion Modelcited by 49
  20. 2026Hierarchical Planning with Latent World Modelscited by 43
  21. 2020Generative Adversarial Networks for Spatio-temporal Data: A Surveycited by 35
  22. 2022A Comprehensive Survey of Regression Based Loss Functions for Time Series Forecastingcited by 33
  23. 2024Scaling Law for Time Series Forecastingcited by 33
  24. 2024Towards Neural Scaling Laws for Time Series Foundation Modelscited by 33
  25. 2024OpenCity: Open Spatio-Temporal Foundation Models for Traffic Predictioncited by 27
  26. 2025Foundation Models for Spatio-Temporal Data Science: A Tutorial and Surveycited by 24
  27. 2023Towards Unifying Diffusion Models for Probabilistic Spatio-Temporal Graph Learningcited by 20
  28. 2024Scaling-laws for Large Time-series Modelscited by 18
  29. 2024Spatio-Temporal Field Neural Networks for Air Quality Inferencecited by 12
  30. 2024T-JEPA: A Joint-Embedding Predictive Architecture for Trajectory Similarity Computationcited by 12
  31. 2026Representation Learning for Spatiotemporal Physical Systemscited by 11
  32. 2025Unraveling Spatio-Temporal Foundation Models via the Pipeline Lens: A Comprehensive Reviewcited by 10
  33. 2025Causal Time Series Generation via Diffusion Modelscited by 8
  34. 2026STReasoner: Empowering LLMs for Spatio-Temporal Reasoning in Time Series via Spatial-Aware Reinforcement Learningcited by 7
  35. 2025Spatio-Temporal Foundation Models: Vision, Challenges, and Opportunitiescited by 6
  36. 2025HiT-JEPA: A Hierarchical Self-supervised Trajectory Embedding Framework for Similarity Computationcited by 6
  37. 2025Synthetic Datasets for Machine Learning on Spatio-Temporal Graphs using PDEscited by 3
  38. 2024Predicting Parking Availability in Singapore with Cross-Domain Data: A New Dataset and A Data-Driven Approachcited by 1

50 most cited of 51 All 51 (JSON)

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2024On Foundation Models for Dynamical Systems from Purely Synthetic Data
  2. 2024FACTS: A Factored State-Space Framework For World Modelling
  3. 2024Causal Deciphering and Inpainting in Spatio-Temporal Dynamics via Diffusion Model
  4. 2026GeoCore-9B: Towards Geo-Aware Generative Foundation Models in Earth Observation
  5. 2025GeoGrid-Bench: Can Foundation Models Understand Multimodal Gridded Geo-Spatial Data?
  6. 2025Towards Reasoning for PDE Foundation Models: A Reward-Model-Driven Inference-Time-Scaling Algorithm
  7. 2026GLIDE: Graph-guided Leap Inference for Diffusion Estimation of Spatio-Temporal Point Processes
  8. 2025FLEX: A Backbone for Diffusion-Based Modeling of Spatio-temporal Physical Systems