# Panda: A pretrained forecast model for chaotic dynamics

arXiv: https://arxiv.org/abs/2505.13755 · HTML: https://mlsanity.com/paper/2505.13755/

## Key facts

- Cited by 13 papers in the map (data through 14 Oct 2026).
- Builds on 49 of its 88 references that are in the map.
- First posted on arXiv 19 May 2025.

## Builds on (most cited first, 30 of 49)

- [Scaling Laws for Neural Language Models](https://mlsanity.com/paper/2001.08361/) (2020)
- [RoFormer: Enhanced Transformer with Rotary Position Embedding](https://mlsanity.com/paper/2104.09864/) (2021)
- [Neural Ordinary Differential Equations](https://mlsanity.com/paper/1806.07366/) (2018)
- [Gaussian Error Linear Units (GELUs)](https://mlsanity.com/paper/1606.08415/) (2016)
- [Efficiently Modeling Long Sequences with Structured State Spaces](https://mlsanity.com/paper/2111.00396/) (2021)
- [A Time Series is Worth 64 Words: Long-term Forecasting with Transformers](https://mlsanity.com/paper/2211.14730/) (2022)
- [Root Mean Square Layer Normalization](https://mlsanity.com/paper/1910.07467/) (2019)
- [Quantifying Attention Flow in Transformers](https://mlsanity.com/paper/2005.00928/) (2020)
- [Chronos: Learning the Language of Time Series](https://mlsanity.com/paper/2403.07815/) (2024)
- [A decoder-only foundation model for time-series forecasting](https://mlsanity.com/paper/2310.10688/) (2023)
- [DeepXDE: A deep learning library for solving differential equations](https://mlsanity.com/paper/1907.04502/) (2019)
- [Neural Operator: Learning Maps Between Function Spaces](https://mlsanity.com/paper/2108.08481/) (2021)
- [Time-MoE: Billion-Scale Time Series Foundation Models with Mixture of Experts](https://mlsanity.com/paper/2409.16040/) (2024)
- [Modern Koopman Theory for Dynamical Systems](https://mlsanity.com/paper/2102.12086/) (2021)
- [Neural General Circulation Models for Weather and Climate](https://mlsanity.com/paper/2311.07222/) (2023)
- [RoboGen: Towards Unleashing Infinite Data for Automated Robot Learning via Generative Simulation](https://mlsanity.com/paper/2311.01455/) (2023)
- [Contemporary Symbolic Regression Methods and their Relative Performance](https://mlsanity.com/paper/2107.14351/) (2021)
- [Poseidon: Efficient Foundation Models for PDEs](https://mlsanity.com/paper/2405.19101/) (2024)
- [Think Globally, Act Locally: A Deep Neural Network Approach to High-Dimensional Time Series Forecasting](https://mlsanity.com/paper/1905.03806/) (2019)
- [Next Generation Reservoir Computing](https://mlsanity.com/paper/2106.07688/) (2021)
- [Towards Foundation Models for Scientific Machine Learning: Characterizing Scaling and Transfer Behavior](https://mlsanity.com/paper/2306.00258/) (2023)
- [Round and Round We Go! What makes Rotary Positional Encodings useful?](https://mlsanity.com/paper/2410.06205/) (2024)
- [Chaos as an interpretable benchmark for forecasting and data-driven modelling](https://mlsanity.com/paper/2110.05266/) (2021)
- [On the difficulty of learning chaotic dynamics with RNNs](https://mlsanity.com/paper/2110.07238/) (2021)
- [ODEFormer: Symbolic Regression of Dynamical Systems with Transformers](https://mlsanity.com/paper/2310.05573/) (2023)
- [Generalized Teacher Forcing for Learning Chaotic Dynamics](https://mlsanity.com/paper/2306.04406/) (2023)
- [Towards a Foundation Model for Partial Differential Equations: Multi-Operator Learning and Extrapolation](https://mlsanity.com/paper/2404.12355/) (2024)
- [Learning Dissipative Dynamics in Chaotic Systems](https://mlsanity.com/paper/2106.06898/) (2021)
- [Identifying nonlinear dynamical systems via generative recurrent neural networks with applications to fMRI](https://mlsanity.com/paper/1902.07186/) (2019)
- [Out-of-Domain Generalization in Dynamical Systems Reconstruction](https://mlsanity.com/paper/2402.18377/) (2024)

## Built on by (most cited first, 13 of 13)

- [Towards mechanistic understanding in a data-driven weather model: internal activations reveal interpretable physical features](https://mlsanity.com/paper/2512.24440/) (2025)
- [Understanding Transformers for Time Series: Rank Structure, Flow-of-ranks, and Compressibility](https://mlsanity.com/paper/2510.03358/) (2025)
- [The Seismic Wavefield Common Task Framework](https://mlsanity.com/paper/2512.19927/) (2025)
- [CTF4Nuclear: Common Task Framework for Nuclear Fission and Fusion Models](https://mlsanity.com/paper/2605.15549/) (2026)
- [Toward World Modeling of Physiological Signals with Chaos-Theoretic Balancing and Latent Dynamics](https://mlsanity.com/paper/2605.15465/) (2026)
- [Minimal Deterministic Echo State Networks Outperform Random Reservoirs in Learning Chaotic Dynamics](https://mlsanity.com/paper/2507.06050/) (2025)
- [Information bottleneck for learning the phase space of dynamics from high-dimensional experimental data](https://mlsanity.com/paper/2604.24662/) (2026)
- [The Dynamic-Probabilistic Consistency Gap in Chaotic Surrogate Modeling](https://mlsanity.com/paper/2605.31547/) (2026)
- [Scaling Law for Large-Scale Pre-Training Using Chaotic Time Series and Predictability in Financial Time Series](https://mlsanity.com/paper/2509.04921/) (2025)
- [Storage and selection of multiple chaotic attractors in minimal reservoir computers](https://mlsanity.com/paper/2603.15155/) (2026)
- [Learning Transferable Predictability Representations](https://mlsanity.com/paper/2605.30592/) (2026)
- [AugRelNet: Relation-Augmented Dynamics for Structure Discovery and Forecasting from Limited Data](https://mlsanity.com/paper/2606.11251/) (2026)
- [Test-Time Forecasting: Recurrent Depth Enables In-Context Dynamics Prediction](https://mlsanity.com/paper/2610.19552/) (2026)

## Abstract (arXiv)

Chaotic systems are intrinsically sensitive to small errors, challenging efforts to construct predictive data-driven models of real-world dynamical systems such as fluid flows or neuronal activity. Prior efforts comprise either specialized models trained on individual time series, or foundation models trained on vast time series databases with little underlying dynamical structure. Motivated by dynamical systems theory, we present Panda, Patched Attention for Nonlinear DynAmics. We train Panda on a novel synthetic, extensible dataset of 2 × 104 chaotic dynamical systems that we discover using an evolutionary algorithm. Trained purely on simulated data, Panda exhibits emergent properties: zero-shot forecasting of unseen chaotic systems preserving both short-term accuracy and distributional measures, nonlinear resonance patterns in attention heads, and effective prediction of real-world experimental time series. Despite having been trained only on low-dimensional ordinary differential equations, Panda spontaneously develops the ability to predict partial differential equations without retraining. We also demonstrate a neural scaling law for differential equations, underscoring the potential of pre-trained models for probing abstract mathematical domains like nonlinear dynamics.