Faster Training of Neural ODEs Using Gauss-Legendre Quadrature
Not on the map. Why?
Abstract (arXiv)
Neural ODEs demonstrate strong performance in generative and time-series modelling. However, training them via the adjoint method is slow compared to discrete models due to the requirement of numerically solving ODEs. To speed neural ODEs up, a common approach is to regularise the solutions. However, this approach may affect the expressivity of the model; when the trajectory itself matters, this is particularly important. In this paper, we propose an alternative way to speed up the training of neural ODEs. The key idea is to speed up the adjoint method by using Gauss-Legendre quadrature to solve integrals faster than ODE-based methods while remaining memory efficient. We also extend the idea to training SDEs using the Wong-Zakai theorem, by training a corresponding ODE and transferring the parameters. Our approach leads to faster training of neural ODEs, especially for large models. It also presents a new way to train SDE-based models.
Key facts. Cited by 3 papers in the map (data through 14 Oct 2026). Builds on 34 of its 53 references that are in the map. First posted on arXiv 21 Aug 2023.
Builds on
34 of 53 references.
References · 34
- 2015Deep Residual Learning for Image Recognitioncited by 52,096
- 2014Adam: A Method for Stochastic Optimizationcited by 37,240
- 2020Score-Based Generative Modeling through Stochastic Differential Equationscited by 9,122
- 2018Neural Ordinary Differential Equationscited by 3,779
- 2015Variational Inference with Normalizing Flowscited by 2,170
- 2021Score-based Generative Modeling in Latent Spacecited by 636
- 2018FFJORD: Free-form Continuous Dynamics for Scalable Reversible Generative Modelscited by 594
- 2020Neural Controlled Differential Equations for Irregular Time Seriescited by 506
- 2019Augmented Neural ODEscited by 397
- 2020Universal Differential Equations for Scientific Machine Learningcited by 353
- 2020Scalable Gradients for Stochastic Differential Equationscited by 271
- 2020How to train your neural ODE: the world of Jacobian and kinetic regularizationcited by 228
22 more
- 2019Latent ODEs for Irregularly-Sampled Time Seriescited by 183
- 2019Neural Stochastic Differential Equations: Deep Latent Gaussian Models in the Diffusion Limitcited by 176
- 2021Neural SDEs as Infinite-Dimensional GANscited by 161
- 2020Dissecting Neural ODEscited by 153
- 2020OT-Flow: Fast and Accurate Continuous Normalizing Flows via Optimal Transportcited by 143
- 2019Neural SDE: Stabilizing Neural ODE Networks with Stochastic Noisecited by 101
- 2019ANODE: Unconditionally Accurate Memory-Efficient Gradients for Neural ODEscited by 84
- 2019On Robustness of Neural Ordinary Differential Equationscited by 83
- 2020Learning Differential Equations that are Easy to Solvecited by 81
- 2020Adaptive Checkpoint Adjoint Method for Gradient Estimation in Neural ODEcited by 76
- 2021Efficient and Accurate Gradients for Neural SDEscited by 71
- 2020On Second Order Behaviour in Augmented Neural ODEscited by 68
- 2020STEER: Simple Temporal Regularization For Neural ODEscited by 58
- 2021Neural ODE Processescited by 57
- 2020Bayesian Neural Ordinary Differential Equationscited by 45
- 2021MALI: A memory efficient and reverse accurate integrator for Neural ODEscited by 42
- 2021Heavy Ball Neural Ordinary Differential Equationscited by 42
- 2021Infinitely Deep Bayesian Neural Networks with Stochastic Differential Equationscited by 38
- 2020"Hey, that's not an ODE": Faster ODE Adjoints via Seminormscited by 35
- 2021Opening the Blackbox: Accelerating Neural Differential Equations by Regularizing Internal Solver Heuristicscited by 26
- 2021Accelerating Neural ODEs Using Model Order Reductioncited by 9
- 2023Correcting Auto-Differentiation in Neural-ODE Trainingcited by 3
Built on by
Citing papers per year: 2025: 1; 2026: 2.
3 papers.