# Learning Differential Equations that are Easy to Solve

arXiv: https://arxiv.org/abs/2007.04504 · HTML: https://mlsanity.com/paper/2007.04504/

## Key facts

- Cited by 81 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 9 Jul 2020.

## Built on by (most cited first, 30 of 81)

- [Flow Matching for Generative Modeling](https://mlsanity.com/paper/2210.02747/) (2022)
- [GRAND: Graph Neural Diffusion](https://mlsanity.com/paper/2106.10934/) (2021)
- [GENIE: Higher-Order Denoising Diffusion Solvers](https://mlsanity.com/paper/2210.05475/) (2022)
- [Neural Flows: Efficient Alternative to Neural ODEs](https://mlsanity.com/paper/2110.13040/) (2021)
- [Minimizing Trajectory Curvature of ODE-based Generative Models](https://mlsanity.com/paper/2301.12003/) (2023)
- [Beltrami Flow and Neural Diffusion on Graphs](https://mlsanity.com/paper/2110.09443/) (2021)
- [Stabilizing Equilibrium Models by Jacobian Regularization](https://mlsanity.com/paper/2106.14342/) (2021)
- [Deep Equilibrium Optical Flow Estimation](https://mlsanity.com/paper/2204.08442/) (2022)
- [Neural Flow Diffusion Models: Learnable Forward Process for Improved Diffusion Modelling](https://mlsanity.com/paper/2404.12940/) (2024)
- [Heavy Ball Neural Ordinary Differential Equations](https://mlsanity.com/paper/2110.04840/) (2021)
- [Neural Networks with Physics-Informed Architectures and Constraints for Dynamical Systems Modeling](https://mlsanity.com/paper/2109.06407/) (2021)
- [Infinitely Deep Bayesian Neural Networks with Stochastic Differential Equations](https://mlsanity.com/paper/2102.06559/) (2021)
- [Constructing Neural Network-Based Models for Simulating Dynamical Systems](https://mlsanity.com/paper/2111.01495/) (2021)
- [Physics-Informed Machine Learning for Modeling and Control of Dynamical Systems](https://mlsanity.com/paper/2306.13867/) (2023)
- [Opening the Blackbox: Accelerating Neural Differential Equations by Regularizing Internal Solver Heuristics](https://mlsanity.com/paper/2105.03918/) (2021)
- [GECCO: Geometrically-Conditioned Point Diffusion Models](https://mlsanity.com/paper/2303.05916/) (2023)
- [Scaling physics-informed hard constraints with mixture-of-experts](https://mlsanity.com/paper/2402.13412/) (2024)
- [Framing RNN as a kernel method: A neural ODE approach](https://mlsanity.com/paper/2106.01202/) (2021)
- [Neural Integral Equations](https://mlsanity.com/paper/2209.15190/) (2022)
- [SDE Matching: Scalable and Simulation-Free Training of Latent Stochastic Differential Equations](https://mlsanity.com/paper/2502.02472/) (2025)
- [Online Deep Equilibrium Learning for Regularization by Denoising](https://mlsanity.com/paper/2205.13051/) (2022)
- [Generalization bounds for neural ordinary differential equations and deep residual networks](https://mlsanity.com/paper/2305.06648/) (2023)
- [Comprehensive Review of Neural Differential Equations for Time Series Analysis](https://mlsanity.com/paper/2502.09885/) (2025)
- [Amortized Reparametrization: Efficient and Scalable Variational Inference for Latent SDEs](https://mlsanity.com/paper/2312.10550/) (2023)
- [ACE-NODE: Attentive Co-Evolving Neural Ordinary Differential Equations](https://mlsanity.com/paper/2105.14953/) (2021)
- [Stabilized Neural Differential Equations for Learning Dynamics with Explicit Constraints](https://mlsanity.com/paper/2306.09739/) (2023)
- [Homotopy-based training of NeuralODEs for accurate dynamics discovery](https://mlsanity.com/paper/2210.01407/) (2022)
- [Taylor-Lagrange Neural Ordinary Differential Equations: Toward Fast Training and Evaluation of Neural ODEs](https://mlsanity.com/paper/2201.05715/) (2022)
- [Second-Order Neural ODE Optimizer](https://mlsanity.com/paper/2109.14158/) (2021)
- [Nuclear Norm Regularization for Deep Learning](https://mlsanity.com/paper/2405.14544/) (2024)

## Abstract (arXiv)

Differential equations parameterized by neural networks become expensive to solve numerically as training progresses. We propose a remedy that encourages learned dynamics to be easier to solve. Specifically, we introduce a differentiable surrogate for the time cost of standard numerical solvers, using higher-order derivatives of solution trajectories. These derivatives are efficient to compute with Taylor-mode automatic differentiation. Optimizing this additional objective trades model performance against the time cost of solving the learned dynamics. We demonstrate our approach by training substantially faster, while nearly as accurate, models in supervised classification, density estimation, and time-series modelling tasks.