mlsanityData to 14 Oct 2026

Panda: A pretrained forecast model for chaotic dynamics

arXiv 2505.13755 ↗First posted 19 May 2025cs.LG cs.NE nlin.CDcited by 13

Not on the map. Why?

Abstract (arXiv)

Chaotic systems are intrinsically sensitive to small errors, challenging efforts to construct predictive data-driven models of real-world dynamical systems such as fluid flows or neuronal activity. Prior efforts comprise either specialized models trained on individual time series, or foundation models trained on vast time series databases with little underlying dynamical structure. Motivated by dynamical systems theory, we present Panda, Patched Attention for Nonlinear DynAmics. We train Panda on a novel synthetic, extensible dataset of 2 × 104 chaotic dynamical systems that we discover using an evolutionary algorithm. Trained purely on simulated data, Panda exhibits emergent properties: zero-shot forecasting of unseen chaotic systems preserving both short-term accuracy and distributional measures, nonlinear resonance patterns in attention heads, and effective prediction of real-world experimental time series. Despite having been trained only on low-dimensional ordinary differential equations, Panda spontaneously develops the ability to predict partial differential equations without retraining. We also demonstrate a neural scaling law for differential equations, underscoring the potential of pre-trained models for probing abstract mathematical domains like nonlinear dynamics.

Key facts. Cited by 13 papers in the map (data through 14 Oct 2026). Builds on 49 of its 88 references that are in the map. First posted on arXiv 19 May 2025.

Builds on

49 of 88 references.

References · 49

  1. 2020Scaling Laws for Neural Language Modelscited by 6,586
  2. 2021RoFormer: Enhanced Transformer with Rotary Position Embeddingcited by 4,703
  3. 2018Neural Ordinary Differential Equationscited by 3,779
  4. 2016Gaussian Error Linear Units (GELUs)cited by 3,502
  5. 2021Efficiently Modeling Long Sequences with Structured State Spacescited by 2,465
  6. 2022A Time Series is Worth 64 Words: Long-term Forecasting with Transformerscited by 2,039
  7. 2019Root Mean Square Layer Normalizationcited by 1,480
  8. 2020Quantifying Attention Flow in Transformerscited by 775
  9. 2024Chronos: Learning the Language of Time Seriescited by 759
  10. 2023A decoder-only foundation model for time-series forecastingcited by 692
  11. 2019DeepXDE: A deep learning library for solving differential equationscited by 619
  12. 2021Neural Operator: Learning Maps Between Function Spacescited by 414
37 more
  1. 2024Time-MoE: Billion-Scale Time Series Foundation Models with Mixture of Expertscited by 287
  2. 2021Modern Koopman Theory for Dynamical Systemscited by 276
  3. 2023Neural General Circulation Models for Weather and Climatecited by 239
  4. 2023RoboGen: Towards Unleashing Infinite Data for Automated Robot Learning via Generative Simulationcited by 218
  5. 2021Contemporary Symbolic Regression Methods and their Relative Performancecited by 215
  6. 2024Poseidon: Efficient Foundation Models for PDEscited by 200
  7. 2019Think Globally, Act Locally: A Deep Neural Network Approach to High-Dimensional Time Series Forecastingcited by 161
  8. 2021Next Generation Reservoir Computingcited by 160
  9. 2023Towards Foundation Models for Scientific Machine Learning: Characterizing Scaling and Transfer Behaviorcited by 131
  10. 2024Round and Round We Go! What makes Rotary Positional Encodings useful?cited by 115
  11. 2021Chaos as an interpretable benchmark for forecasting and data-driven modellingcited by 88
  12. 2021On the difficulty of learning chaotic dynamics with RNNscited by 67
  13. 2023ODEFormer: Symbolic Regression of Dynamical Systems with Transformerscited by 62
  14. 2023Generalized Teacher Forcing for Learning Chaotic Dynamicscited by 58
  15. 2024Towards a Foundation Model for Partial Differential Equations: Multi-Operator Learning and Extrapolationcited by 46
  16. 2021Learning Dissipative Dynamics in Chaotic Systemscited by 42
  17. 2019Identifying nonlinear dynamical systems via generative recurrent neural networks with applications to fMRIcited by 41
  18. 2024Out-of-Domain Generalization in Dynamical Systems Reconstructioncited by 33
  19. 2024LLMs learn governing principles of dynamical systems, revealing an in-context neural scaling lawcited by 32
  20. 2024Data-Efficient Operator Learning via Unsupervised Pretraining and In-Context Learningcited by 32
  21. 2024Zero-shot forecasting of chaotic systemscited by 28
  22. 2023Model scale versus domain knowledge in statistical forecasting of chaotic systemscited by 27
  23. 2023Predicting Ordinary Differential Equations with Transformerscited by 26
  24. 2025Thermalizer: Stable autoregressive neural emulation of spatiotemporal chaoscited by 21
  25. 2024Automating the Search for Artificial Life with Foundation Modelscited by 20
  26. 2024FMint: Bridging Human Designed and Data Pretrained Models for Differential Equation Foundation Modelcited by 19
  27. 2024Almost-Linear RNNs Yield Highly Interpretable Symbolic Codes in Dynamical Systems Reconstructioncited by 19
  28. 2025Context parroting: A simple but tough-to-beat baseline for foundation models in scientific machine learningcited by 18
  29. 2022Data Augmentation vs. Equivariant Networks: A Theory of Generalization on Dynamics Forecastingcited by 17
  30. 2019Intrinsically Motivated Discovery of Diverse Patterns in Self-Organizing Systemscited by 13
  31. 2024Zero-shot Imputation with Foundation Inference Models for Dynamical Systemscited by 13
  32. 2023A foundational neural operator that continuously learns without forgettingcited by 11
  33. 2023Inferring dynamic regulatory interaction graphs from time series data with perturbationscited by 9
  34. 2023Recurrences reveal shared causal drivers of complex time seriescited by 5
  35. 2024Decision Transformer as a Foundation Model for Partially Observable Continuous Controlcited by 5
  36. 2024On Foundation Models for Dynamical Systems from Purely Synthetic Datacited by 4
  37. 2025Proc4Gem: Foundation models for physical agency through procedural generationcited by 2

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 5; 2026: 8.

13 papers.

Sort:

By area:

  1. 2025Towards mechanistic understanding in a data-driven weather model: internal activations reveal interpretable physical featurescited by 12
  2. 2025Understanding Transformers for Time Series: Rank Structure, Flow-of-ranks, and Compressibilitycited by 9
  3. 2025The Seismic Wavefield Common Task Frameworkcited by 6
  4. 2026CTF4Nuclear: Common Task Framework for Nuclear Fission and Fusion Modelscited by 5
  5. 2026Toward World Modeling of Physiological Signals with Chaos-Theoretic Balancing and Latent Dynamicscited by 3
  6. 2025Minimal Deterministic Echo State Networks Outperform Random Reservoirs in Learning Chaotic Dynamicscited by 2
  7. 2026Information bottleneck for learning the phase space of dynamics from high-dimensional experimental datacited by 1
  8. 2026The Dynamic-Probabilistic Consistency Gap in Chaotic Surrogate Modelingcited by 1
  9. 2025Scaling Law for Large-Scale Pre-Training Using Chaotic Time Series and Predictability in Financial Time Seriescited by 0
  10. 2026Storage and selection of multiple chaotic attractors in minimal reservoir computerscited by 0
  11. 2026Learning Transferable Predictability Representationscited by 0
  12. 2026AugRelNet: Relation-Augmented Dynamics for Structure Discovery and Forecasting from Limited Datacited by 0
1 more
  1. 2026Test-Time Forecasting: Recurrent Depth Enables In-Context Dynamics Predictioncited by 0