Correcting Auto-Differentiation in Neural-ODE Training
Not on the map. Why?
Abstract (arXiv)
Does the use of auto-differentiation yield reasonable updates for deep neural networks (DNNs)? Specifically, when DNNs are designed to adhere to neural ODE architectures, can we trust the gradients provided by auto-differentiation? Through mathematical analysis and numerical evidence, we demonstrate that when neural networks employ high-order methods, such as Linear Multistep Methods (LMM) or Explicit Runge-Kutta Methods (ERK), to approximate the underlying ODE flows, brute-force auto-differentiation often introduces artificial oscillations in the gradients that prevent convergence. In the case of Leapfrog and 2-stage ERK, we propose simple post-processing techniques that effectively eliminates these oscillations, correct the gradient computation and thus returns the accurate updates.
Key facts. Cited by 1 paper in the map (data through 7 Oct 2026). Builds on 20 of its 32 references that are in the map. First posted on arXiv 3 Jun 2023.
Builds on
20 of 32 references.
References · 20
- 2018Neural Ordinary Differential Equationscited by 3,733
- 2019Normalizing Flows for Probabilistic Modeling and Inferencecited by 1,315
- 2019Augmented Neural ODEscited by 396
- 2019Machine Learning for Fluid Mechanicscited by 380
- 2020Universal Differential Equations for Scientific Machine Learningcited by 345
- 2017Beyond Finite Layer Neural Networks: Bridging Deep Architectures and Numerical Differential Equationscited by 203
- 2019Symplectic Recurrent Neural Networkscited by 148
- 2019ANODE: Unconditionally Accurate Memory-Efficient Gradients for Neural ODEscited by 83
- 2017Reversible Architectures for Arbitrarily Deep Residual Neural Networkscited by 69
- 2020A Mean-field Analysis of Deep ResNet and Beyond: Towards Provable Optimization Via Overparameterization From Depthcited by 56
- 2020Scaling Hamiltonian Monte Carlo Inference for Bayesian Neural Networks with Symmetric Splittingcited by 54
- 2020Discretize-Optimize vs. Optimize-Discretize for Time-Series Regression and Continuous Normalizing Flowscited by 36
8 more
- 2021Overparameterization of deep ResNet: zero loss and mean-field analysiscited by 24
- 2020Exchangeable Neural ODE for Set Modelingcited by 19
- 2021Forecasting the outcome of spintronic experiments with Neural Ordinary Differential Equationscited by 13
- 2021On the Global Convergence of Gradient Descent for multi-layer ResNets in the mean-field regimecited by 10
- 2022Learning Differential Operators for Interpretable Time Series Modelingcited by 7
- 2017Non-commutative Discretize-then-Optimize Algorithms for Elliptic PDE-Constrained Optimal Control Problemscited by 2
- 2020Forecasting Black Sigatoka Infection Risks with Latent Neural ODEscited by 1
- 2023Understanding Automatic Differentiation Pitfallscited by 1
Built on by
Citing papers per year: 2023: 1.
1 paper.