mlsanityData to 14 Oct 2026

On Foundation Models for Dynamical Systems from Purely Synthetic Data

arXiv 2412.00395 ↗First posted 30 Nov 2024cs.LG cs.RO stat.MLcited by 4

On the map: Synthetic data and self-improvement for LLMs → Model collapse from synthetic data

Abstract (arXiv)

Foundation models have demonstrated remarkable generalization, data efficiency, and robustness properties across various domains. In this paper, we explore the feasibility of foundation models for applications in the control domain. The success of these models is enabled by large-scale pretaining on Internet-scale datasets. These are available in fields like natural language processing and computer vision, but do not exist for dynamical systems. We address this challenge by pretraining a transformer-based foundation model exclusively on synthetic data and propose to sample dynamics functions from a reproducing kernel Hilbert space. Our pretrained model generalizes for prediction tasks across different dynamical systems, which we validate in simulation and hardware experiments, including cart-pole and Furuta pendulum setups. Additionally, the model can be fine-tuned effectively to new systems to increase performance even further. Our results demonstrate the feasibility of foundation models for dynamical systems that outperform specialist models in terms of generalization, data efficiency, and robustness.

Key facts. Cited by 4 papers in the map (data through 14 Oct 2026). Placed in Model collapse from synthetic data, one of 13 directions in Synthetic data and self-improvement for LLMs. Builds on 32 of its 41 references that are in the map. First posted on arXiv 30 Nov 2024.

Builds on

32 of 41 references.

References · 32

  1. 2017Attention Is All You Needcited by 60,361
  2. 2020Language Models are Few-Shot Learnerscited by 32,419
  3. 2023Segment Anythingcited by 8,234
  4. 2023Grounding DINO: Marrying DINO with Grounded Pre-Training for Open-Set Object Detectioncited by 3,332
  5. 2022RT-1: Robotics Transformer for Real-World Control at Scalecited by 2,061
  6. 2022A Time Series is Worth 64 Words: Long-term Forecasting with Transformerscited by 2,039
  7. 2022Are Transformers Effective for Time Series Forecasting?cited by 1,424
  8. 2023Open X-Embodiment: Robotic Learning Datasets and RT-X Modelscited by 862
  9. 2023A decoder-only foundation model for time-series forecastingcited by 692
  10. 2020It's Not Just Size That Matters: Small Language Models Are Also Few-Shot Learnerscited by 669
  11. 2019Using Pre-Training Can Improve Model Robustness and Uncertaintycited by 373
  12. 2023A Comprehensive Survey on Pretrained Foundation Models: A History from BERT to ChatGPTcited by 368
20 more
  1. 2023Long-term Forecasting with TiDE: Time-series Dense Encodercited by 332
  2. 2020Time Series Data Augmentation for Deep Learning: A Surveycited by 226
  3. 2023ViNT: A Foundation Model for Visual Navigationcited by 222
  4. 2023Foundation Models in Robotics: Applications, Challenges, and the Futurecited by 216
  5. 2024Timer: Generative Pre-trained Transformers Are Large Time Series Modelscited by 208
  6. 2018Gaussian Processes and Kernel Methods: A Review on Connections and Equivalencescited by 192
  7. 2023TimeGPT-1cited by 158
  8. 2023Large Language Models for Robotics: A Surveycited by 133
  9. 2021Practical and Rigorous Uncertainty Bounds for Gaussian Process Regressioncited by 49
  10. 2021Reproducing Kernel Hilbert Space, Mercer's Theorem, Eigenfunctions, Nystr"om Method, and Use of Kernels in Machine Learning: Tutorial and Surveycited by 37
  11. 2022On the Power of Foundation Modelscited by 37
  12. 2023Optimistic Active Exploration of Dynamical Systemscited by 32
  13. 2024Few-shot Adaptation of Multi-modal Foundation Models: A Surveycited by 28
  14. 2019Actively Learning Gaussian Process Dynamicscited by 26
  15. 2023A Generalist Dynamics Model for Controlcited by 24
  16. 2024FMint: Bridging Human Designed and Data Pretrained Models for Differential Equation Foundation Modelcited by 19
  17. 2016A flexible state space model for learning nonlinear dynamical systemscited by 18
  18. 2020Learning Constrained Dynamics with Gauss Principle adhering Gaussian Processescited by 13
  19. 2024Zero-shot Imputation with Foundation Inference Models for Dynamical Systemscited by 13
  20. 2024Parameter-Adaptive Approximate MPC: Tuning Neural-Network Controllers without Retrainingcited by 7

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 3; 2026: 1.

4 papers.

  1. 2025Panda: A pretrained forecast model for chaotic dynamicscited by 13
  2. 2025Learning Dynamics from Input-Output Data with Hamiltonian Gaussian Processescited by 1
  3. 2025Sailing Towards Zero-Shot State Estimation using Foundation Models Combined with a UKFcited by 0
  4. 2026Attenuated in-context identification in time-series foundation models: diagnosis under counterfactual inputs and repair by synthetic forced-system fine-tuningcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026Learning Nonlinear Systems In-Context: From Synthetic Data to Real-World Motor Control
  2. 2026Learning Spatio-Temporal Foundation Models from Pure Synthetic Data
  3. 2026Foundation Models in Robotics: A Comprehensive Review of Methods, Models, Datasets, Challenges and Future Research Directions
  4. 2025What Has a Foundation Model Found? Using Inductive Bias to Probe for World Models
  5. 2025Self-Improving Embodied Foundation Models
  6. 2024HiPPO-Prophecy: State-Space Models can Provably Learn Dynamical Systems in Context
  7. 2024eXponential FAmily Dynamical Systems (XFADS): Large-scale nonlinear Gaussian state-space modeling
  8. 2025Learning State-Space Models of Dynamic Systems from Arbitrary Data using Joint Embedding Predictive Architectures