# Correcting Auto-Differentiation in Neural-ODE Training

arXiv: https://arxiv.org/abs/2306.02192 · HTML: https://mlsanity.com/paper/2306.02192/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Builds on 20 of its 32 references that are in the map.
- First posted on arXiv 3 Jun 2023.

## Builds on (most cited first, 20 of 20)

- [Neural Ordinary Differential Equations](https://mlsanity.com/paper/1806.07366/) (2018)
- [Normalizing Flows for Probabilistic Modeling and Inference](https://mlsanity.com/paper/1912.02762/) (2019)
- [Augmented Neural ODEs](https://mlsanity.com/paper/1904.01681/) (2019)
- [Machine Learning for Fluid Mechanics](https://mlsanity.com/paper/1905.11075/) (2019)
- [Universal Differential Equations for Scientific Machine Learning](https://mlsanity.com/paper/2001.04385/) (2020)
- [Beyond Finite Layer Neural Networks: Bridging Deep Architectures and Numerical Differential Equations](https://mlsanity.com/paper/1710.10121/) (2017)
- [Symplectic Recurrent Neural Networks](https://mlsanity.com/paper/1909.13334/) (2019)
- [ANODE: Unconditionally Accurate Memory-Efficient Gradients for Neural ODEs](https://mlsanity.com/paper/1902.10298/) (2019)
- [Reversible Architectures for Arbitrarily Deep Residual Neural Networks](https://mlsanity.com/paper/1709.03698/) (2017)
- [A Mean-field Analysis of Deep ResNet and Beyond: Towards Provable Optimization Via Overparameterization From Depth](https://mlsanity.com/paper/2003.05508/) (2020)
- [Scaling Hamiltonian Monte Carlo Inference for Bayesian Neural Networks with Symmetric Splitting](https://mlsanity.com/paper/2010.06772/) (2020)
- [Discretize-Optimize vs. Optimize-Discretize for Time-Series Regression and Continuous Normalizing Flows](https://mlsanity.com/paper/2005.13420/) (2020)
- [Overparameterization of deep ResNet: zero loss and mean-field analysis](https://mlsanity.com/paper/2105.14417/) (2021)
- [Exchangeable Neural ODE for Set Modeling](https://mlsanity.com/paper/2008.02676/) (2020)
- [Forecasting the outcome of spintronic experiments with Neural Ordinary Differential Equations](https://mlsanity.com/paper/2108.02318/) (2021)
- [On the Global Convergence of Gradient Descent for multi-layer ResNets in the mean-field regime](https://mlsanity.com/paper/2110.02926/) (2021)
- [Learning Differential Operators for Interpretable Time Series Modeling](https://mlsanity.com/paper/2209.01491/) (2022)
- [Non-commutative Discretize-then-Optimize Algorithms for Elliptic PDE-Constrained Optimal Control Problems](https://mlsanity.com/paper/1706.07652/) (2017)
- [Forecasting Black Sigatoka Infection Risks with Latent Neural ODEs](https://mlsanity.com/paper/2012.00752/) (2020)
- [Understanding Automatic Differentiation Pitfalls](https://mlsanity.com/paper/2305.07546/) (2023)

## Built on by (most cited first, 3 of 3)

- [Faster Training of Neural ODEs Using Gauss-Legendre Quadrature](https://mlsanity.com/paper/2308.10644/) (2023)
- [Zero-Shot Size Transfer for Neural ODEs on Sparse Random Graphs: Graphon Limits and Adjoint Convergence](https://mlsanity.com/paper/2606.26662/) (2026)
- [Branch Geometry and Finite-Radius Sensitivity of Hard-ReLU Training](https://mlsanity.com/paper/2608.30960/) (2026)

## Abstract (arXiv)

Does the use of auto-differentiation yield reasonable updates for deep neural networks (DNNs)? Specifically, when DNNs are designed to adhere to neural ODE architectures, can we trust the gradients provided by auto-differentiation? Through mathematical analysis and numerical evidence, we demonstrate that when neural networks employ high-order methods, such as Linear Multistep Methods (LMM) or Explicit Runge-Kutta Methods (ERK), to approximate the underlying ODE flows, brute-force auto-differentiation often introduces artificial oscillations in the gradients that prevent convergence. In the case of Leapfrog and 2-stage ERK, we propose simple post-processing techniques that effectively eliminates these oscillations, correct the gradient computation and thus returns the accurate updates.