# ODEFormer: Symbolic Regression of Dynamical Systems with Transformers

arXiv: https://arxiv.org/abs/2310.05573 · HTML: https://mlsanity.com/paper/2310.05573/

## Key facts

- Cited by 62 papers in the map (data through 14 Oct 2026).
- Builds on 29 of its 69 references that are in the map.
- First posted on arXiv 9 Oct 2023.

## Builds on (most cited first, 29 of 29)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Neural Ordinary Differential Equations](https://mlsanity.com/paper/1806.07366/) (2018)
- [Mastering Atari, Go, Chess and Shogi by Planning with a Learned Model](https://mlsanity.com/paper/1911.08265/) (2019)
- [Sigmoid-Weighted Linear Units for Neural Network Function Approximation in Reinforcement Learning](https://mlsanity.com/paper/1702.03118/) (2017)
- [AI Feynman: a Physics-Inspired Method for Symbolic Regression](https://mlsanity.com/paper/1905.11481/) (2019)
- [A Survey on Explainable Artificial Intelligence (XAI): Towards Medical XAI](https://mlsanity.com/paper/1907.07374/) (2019)
- [Generative Language Modeling for Automated Theorem Proving](https://mlsanity.com/paper/2009.03393/) (2020)
- [Deep symbolic regression: Recovering mathematical expressions from data via risk-seeking policy gradients](https://mlsanity.com/paper/1912.04871/) (2019)
- [Deep Learning for Symbolic Mathematics](https://mlsanity.com/paper/1912.01412/) (2019)
- [Contemporary Symbolic Regression Methods and their Relative Performance](https://mlsanity.com/paper/2107.14351/) (2021)
- [End-to-end symbolic regression with transformers](https://mlsanity.com/paper/2204.10532/) (2022)
- [SymbolicGPT: A Generative Transformer Model for Symbolic Regression](https://mlsanity.com/paper/2106.14131/) (2021)
- [Interpretable Scientific Discovery with Symbolic Regression: A Review](https://mlsanity.com/paper/2211.10873/) (2022)
- [Chaos as an interpretable benchmark for forecasting and data-driven modelling](https://mlsanity.com/paper/2110.05266/) (2021)
- [Linear algebra with transformers](https://mlsanity.com/paper/2112.01898/) (2021)
- [A Bayesian machine scientist to aid in the solution of challenging scientific problems](https://mlsanity.com/paper/2004.12157/) (2020)
- [SymFormer: End-to-end symbolic regression using transformer-based architecture](https://mlsanity.com/paper/2205.15764/) (2022)
- [Teaching Temporal Logics to Neural Networks](https://mlsanity.com/paper/2003.04218/) (2020)
- [Learning concise representations for regression by evolving networks of trees](https://mlsanity.com/paper/1807.00981/) (2018)
- [Epsilon-Lexicase Selection for Regression](https://mlsanity.com/paper/1905.13266/) (2019)
- [Predicting Ordinary Differential Equations with Transformers](https://mlsanity.com/paper/2307.12617/) (2023)
- [Symbolic Pregression: Discovering Physical Laws from Distorted Video](https://mlsanity.com/paper/2005.11212/) (2020)
- [Data-driven discovery of free-form governing differential equations](https://mlsanity.com/paper/1910.05117/) (2019)
- [GSR: A Generalized Symbolic Regression Approach](https://mlsanity.com/paper/2205.15569/) (2022)
- [Identifiability Analysis of Linear Ordinary Differential Equation Systems with a Single Trajectory](https://mlsanity.com/paper/2103.05660/) (2021)
- [Accelerating Understanding of Scientific Experiments with End to End Symbolic Regression](https://mlsanity.com/paper/2112.04023/) (2021)
- [Inferring the Structure of Ordinary Differential Equations](https://mlsanity.com/paper/2107.07345/) (2021)
- [MMViT: Multiscale Multiview Vision Transformers](https://mlsanity.com/paper/2305.00104/) (2023)

## Built on by (most cited first, 30 of 62)

- [Out-of-Domain Generalization in Dynamical Systems Reconstruction](https://mlsanity.com/paper/2402.18377/) (2024)
- [Marrying Causal Representation Learning with Dynamical Systems for Science](https://mlsanity.com/paper/2405.13888/) (2024)
- [LLM4ED: Large Language Models for Automatic Equation Discovery](https://mlsanity.com/paper/2405.07761/) (2024)
- [Zero-shot Imputation with Foundation Inference Models for Dynamical Systems](https://mlsanity.com/paper/2402.07594/) (2024)
- [Panda: A pretrained forecast model for chaotic dynamics](https://mlsanity.com/paper/2505.13755/) (2025)
- [LLM-AutoSciLab: Closed-Loop Scientific Discovery via Active Experimentation with LLMs](https://mlsanity.com/paper/2605.24043/) (2026)
- [In-Context Learning of Temporal Point Processes with Foundation Inference Models](https://mlsanity.com/paper/2509.24762/) (2025)
- [Data-driven model discovery with Kolmogorov-Arnold networks](https://mlsanity.com/paper/2409.15167/) (2024)
- [Self-supervised contrastive learning performs non-linear system identification](https://mlsanity.com/paper/2410.14673/) (2024)
- [In-Context Learning of Stochastic Differential Equations with Foundation Inference Models](https://mlsanity.com/paper/2502.19049/) (2025)
- [Identifiability Challenges in Sparse Linear Ordinary Differential Equations](https://mlsanity.com/paper/2506.09816/) (2025)
- [MDBench: Benchmarking Data-Driven Methods for Model Discovery](https://mlsanity.com/paper/2509.20529/) (2025)
- [Vertical Symbolic Regression via Deep Policy Gradient](https://mlsanity.com/paper/2402.00254/) (2024)
- [Grammar-based Ordinary Differential Equation Discovery](https://mlsanity.com/paper/2504.02630/) (2025)
- [Can Large Language Models Adequately Perform Symbolic Reasoning Over Time Series?](https://mlsanity.com/paper/2508.03963/) (2025)
- [A Comparison of Recent Algorithms for Symbolic Regression to Genetic Programming](https://mlsanity.com/paper/2406.03585/) (2024)
- [Scalable Mechanistic Neural Networks for Differential Equations and Machine Learning](https://mlsanity.com/paper/2410.06074/) (2024)
- [Adaptive parameters identification for nonlinear dynamics using deep permutation invariant networks](https://mlsanity.com/paper/2501.11350/) (2025)
- [Al-Khwarizmi: Discovering Physical Laws with Foundation Models](https://mlsanity.com/paper/2502.01702/) (2025)
- [Discovering Physics Laws of Dynamical Systems via Invariant Function Learning](https://mlsanity.com/paper/2502.04495/) (2025)
- [Towards Foundational Models for Dynamical System Reconstruction: Hierarchical Meta-Learning via Mixture of Experts](https://mlsanity.com/paper/2502.05335/) (2025)
- [Mechanistic PDE Networks for Discovery of Governing Equations](https://mlsanity.com/paper/2502.18377/) (2025)
- [Position: Biology is the Challenge Physics-Informed ML Needs to Evolve](https://mlsanity.com/paper/2510.25368/) (2025)
- [LLM-ODE: Data-driven Discovery of Dynamical Systems with Large Language Models](https://mlsanity.com/paper/2603.20910/) (2026)
- [Active Symbolic Discovery of Ordinary Differential Equations via Phase Portrait Sketching](https://mlsanity.com/paper/2409.01416/) (2024)
- [Neuronal Attention Circuit (NAC) for Representation Learning](https://mlsanity.com/paper/2512.10282/) (2025)
- [Foundation Inference Models for Ordinary Differential Equations](https://mlsanity.com/paper/2602.08733/) (2026)
- [Data-driven discovery of governing differential equations across physical systems](https://mlsanity.com/paper/2606.09638/) (2026)
- [Neural Symbolic Regression of Complex Network Dynamics](https://mlsanity.com/paper/2410.11185/) (2024)
- [Data-driven Discovery of Digital Twins in Biomedical Research](https://mlsanity.com/paper/2508.21484/) (2025)

## Abstract (arXiv)

We introduce ODEFormer, the first transformer able to infer multidimensional ordinary differential equation (ODE) systems in symbolic form from the observation of a single solution trajectory. We perform extensive evaluations on two datasets: (i) the existing "Strogatz" dataset featuring two-dimensional systems; (ii) ODEBench, a collection of one- to four-dimensional systems that we carefully curated from the literature to provide a more holistic benchmark. ODEFormer consistently outperforms existing methods while displaying substantially improved robustness to noisy and irregularly sampled observations, as well as faster inference. We release our code, model and benchmark dataset publicly.