# Proximal Implicit ODE Solvers for Accelerating Learning Neural ODEs

arXiv: https://arxiv.org/abs/2204.08621 · HTML: https://mlsanity.com/paper/2204.08621/

## Key facts

- Cited by 6 papers in the map (data through 14 Oct 2026).
- Builds on 29 of its 64 references that are in the map.
- First posted on arXiv 19 Apr 2022.

## Builds on (most cited first, 29 of 29)

- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Graph Attention Networks](https://mlsanity.com/paper/1710.10903/) (2017)
- [Neural Ordinary Differential Equations](https://mlsanity.com/paper/1806.07366/) (2018)
- [PointFlow: 3D Point Cloud Generation with Continuous Normalizing Flows](https://mlsanity.com/paper/1906.12320/) (2019)
- [Augmented Neural ODEs](https://mlsanity.com/paper/1904.01681/) (2019)
- [Entropy-SGD: Biasing Gradient Descent Into Wide Valleys](https://mlsanity.com/paper/1611.01838/) (2016)
- [GRAND: Graph Neural Diffusion](https://mlsanity.com/paper/2106.10934/) (2021)
- [How to train your neural ODE: the world of Jacobian and kinetic regularization](https://mlsanity.com/paper/2002.02798/) (2020)
- [Graph Neural Ordinary Differential Equations](https://mlsanity.com/paper/1911.07532/) (2019)
- [Dissecting Neural ODEs](https://mlsanity.com/paper/2002.08071/) (2020)
- [Continuous Graph Neural Networks](https://mlsanity.com/paper/1912.00967/) (2019)
- [ANODE: Unconditionally Accurate Memory-Efficient Gradients for Neural ODEs](https://mlsanity.com/paper/1902.10298/) (2019)
- [Learning Differential Equations that are Easy to Solve](https://mlsanity.com/paper/2007.04504/) (2020)
- [Adaptive Checkpoint Adjoint Method for Gradient Estimation in Neural ODE](https://mlsanity.com/paper/2006.02493/) (2020)
- [On Second Order Behaviour in Augmented Neural ODEs](https://mlsanity.com/paper/2006.07220/) (2020)
- [STEER: Simple Temporal Regularization For Neural ODEs](https://mlsanity.com/paper/2006.10711/) (2020)
- [Heavy Ball Neural Ordinary Differential Equations](https://mlsanity.com/paper/2110.04840/) (2021)
- [Hypersolvers: Toward Fast Continuous-Depth Models](https://mlsanity.com/paper/2007.09601/) (2020)
- [ShapeFlow: Learnable Deformations Among 3D Shapes](https://mlsanity.com/paper/2006.07982/) (2020)
- [Deep Relaxation: partial differential equations for optimizing deep neural networks](https://mlsanity.com/paper/1704.04932/) (2017)
- [Opening the Blackbox: Accelerating Neural Differential Equations by Regularizing Internal Solver Heuristics](https://mlsanity.com/paper/2105.03918/) (2021)
- [Proximal Algorithms in Statistics and Machine Learning](https://mlsanity.com/paper/1502.03175/) (2015)
- [Particle-based Energetic Variational Inference](https://mlsanity.com/paper/2004.06443/) (2020)
- [Symplectic Adjoint Method for Exact Gradient of Neural ODE with Minimal Memory](https://mlsanity.com/paper/2102.09750/) (2021)
- [Sparsity Meets Robustness: Channel Pruning for the Feynman-Kac Formalism Principled Robust Deep Neural Nets](https://mlsanity.com/paper/2003.00631/) (2020)
- [Stochastic Gradient Descent with Nonlinear Conjugate Gradient-Style Adaptive Momentum](https://mlsanity.com/paper/2012.02188/) (2020)
- [From differential equation solvers to accelerated first-order methods for convex optimization](https://mlsanity.com/paper/1909.03145/) (2019)
- [Neural Ordinary Differential Equations for Data-Driven Reduced Order Modeling of Environmental Hydrodynamics](https://mlsanity.com/paper/2104.13962/) (2021)
- [Transformed Schatten-1 Iterative Thresholding Algorithms for Low Rank Matrix Completion](https://mlsanity.com/paper/1506.04444/) (2015)

## Built on by (most cited first, 6 of 6)

- [Training Stiff Neural Ordinary Differential Equations with Implicit Single-Step Methods](https://mlsanity.com/paper/2410.05592/) (2024)
- [Training Stiff Neural Ordinary Differential Equations with Explicit Exponential Integration Methods](https://mlsanity.com/paper/2412.01181/) (2024)
- [The Vanishing Gradient Problem for Stiff Neural Differential Equations](https://mlsanity.com/paper/2508.01519/) (2025)
- [Semi-Implicit Neural Ordinary Differential Equations](https://mlsanity.com/paper/2412.11301/) (2024)
- [Physics-Informed Neural ODEs with Scale-Aware Residuals for Learning Stiff Biophysical Dynamics](https://mlsanity.com/paper/2511.11734/) (2025)
- [Stable CDE Autoencoders with Acuity Regularization for Offline Reinforcement Learning in Sepsis Treatment](https://mlsanity.com/paper/2506.15019/) (2025)

## Abstract (arXiv)

Learning neural ODEs often requires solving very stiff ODE systems, primarily using explicit adaptive step size ODE solvers. These solvers are computationally expensive, requiring the use of tiny step sizes for numerical stability and accuracy guarantees. This paper considers learning neural ODEs using implicit ODE solvers of different orders leveraging proximal operators. The proximal implicit solver consists of inner-outer iterations: the inner iterations approximate each implicit update step using a fast optimization algorithm, and the outer iterations solve the ODE system over time. The proximal implicit ODE solver guarantees superiority over explicit solvers in numerical stability and computational efficiency. We validate the advantages of proximal implicit solvers over existing popular neural ODE solvers on various challenging benchmark tasks, including learning continuous-depth graph neural networks and continuous normalizing flows.