# On Numerical Integration in Neural Ordinary Differential Equations

arXiv: https://arxiv.org/abs/2206.07335 · HTML: https://mlsanity.com/paper/2206.07335/

## Key facts

- Cited by 21 papers in the map (data through 14 Oct 2026).
- Builds on 30 of its 57 references that are in the map.
- First posted on arXiv 15 Jun 2022.

## Builds on (most cited first, 30 of 30)

- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Neural Ordinary Differential Equations](https://mlsanity.com/paper/1806.07366/) (2018)
- [Hamiltonian Neural Networks](https://mlsanity.com/paper/1906.01563/) (2019)
- [Augmented Neural ODEs](https://mlsanity.com/paper/1904.01681/) (2019)
- [Beyond Finite Layer Neural Networks: Bridging Deep Architectures and Numerical Differential Equations](https://mlsanity.com/paper/1710.10121/) (2017)
- [Training behavior of deep neural network in frequency domain](https://mlsanity.com/paper/1807.01251/) (2018)
- [Dissecting Neural ODEs](https://mlsanity.com/paper/2002.08071/) (2020)
- [Neural Jump Stochastic Differential Equations](https://mlsanity.com/paper/1905.10403/) (2019)
- [Multistep Neural Networks for Data-driven Discovery of Nonlinear Dynamical Systems](https://mlsanity.com/paper/1801.01236/) (2018)
- [On Robustness of Neural Ordinary Differential Equations](https://mlsanity.com/paper/1910.05513/) (2019)
- [Maximum Principle Based Algorithms for Deep Learning](https://mlsanity.com/paper/1710.09513/) (2017)
- [Adaptive Checkpoint Adjoint Method for Gradient Estimation in Neural ODE](https://mlsanity.com/paper/2006.02493/) (2020)
- [A Mean-Field Optimal Control Formulation of Deep Learning](https://mlsanity.com/paper/1807.01083/) (2018)
- [Neural ODE Processes](https://mlsanity.com/paper/2103.12413/) (2021)
- [Approximation Capabilities of Neural ODEs and Invertible Residual Networks](https://mlsanity.com/paper/1907.12998/) (2019)
- [Theory of the Frequency Principle for General Deep Neural Networks](https://mlsanity.com/paper/1906.09235/) (2019)
- [Heavy Ball Neural Ordinary Differential Equations](https://mlsanity.com/paper/2110.04840/) (2021)
- [Data-driven Prediction of General Hamiltonian Dynamics via Learning Exactly-Symplectic Maps](https://mlsanity.com/paper/2103.05632/) (2021)
- [Continuous-in-Depth Neural Networks](https://mlsanity.com/paper/2008.02389/) (2020)
- [OnsagerNet: Learning Stable and Interpretable Dynamics using a Generalized Onsager Principle](https://mlsanity.com/paper/2009.02327/) (2020)
- [ODE2VAE: Deep generative second order ODEs with Bayesian neural networks](https://mlsanity.com/paper/1905.10994/) (2019)
- [Hypersolvers: Toward Fast Continuous-Depth Models](https://mlsanity.com/paper/2007.09601/) (2020)
- [Learning continuous models for continuous physics](https://mlsanity.com/paper/2202.08494/) (2022)
- [Opening the Blackbox: Accelerating Neural Differential Equations by Regularizing Internal Solver Heuristics](https://mlsanity.com/paper/2105.03918/) (2021)
- [On dissipative symplectic integration with applications to gradient-based optimization](https://mlsanity.com/paper/2004.06840/) (2020)
- [ResNet After All? Neural ODEs and Their Numerical Solution](https://mlsanity.com/paper/2007.15386/) (2020)
- [Time-Reversal Symmetric ODE Network](https://mlsanity.com/paper/2007.11362/) (2020)
- [Discovery of Dynamics Using Linear Multistep Methods](https://mlsanity.com/paper/1912.12728/) (2019)
- [The Discovery of Dynamics via Linear Multistep Methods and Deep Learning: Error Estimation](https://mlsanity.com/paper/2103.11488/) (2021)
- [Towards Understanding Normalization in Neural ODEs](https://mlsanity.com/paper/2004.09222/) (2020)

## Built on by (most cited first, 21 of 21)

- [SEGNO: Generalizing Equivariant Graph Neural Networks with Physical Inductive Biases](https://mlsanity.com/paper/2308.13212/) (2023)
- [Bayesian Gaussian Process ODEs via Double Normalizing Flows](https://mlsanity.com/paper/2309.09222/) (2023)
- [Implementation and (Inverse Modified) Error Analysis for implicitly-templated ODE-nets](https://mlsanity.com/paper/2303.17824/) (2023)
- [Stability-Informed Initialization of Neural Ordinary Differential Equations](https://mlsanity.com/paper/2311.15890/) (2023)
- [SeqLink: A Robust Neural-ODE Architecture for Modelling Partially Observed Time Series](https://mlsanity.com/paper/2212.03560/) (2022)
- [Some of the variables, some of the parameters, some of the times, with some physics known: Identification with partial information](https://mlsanity.com/paper/2304.14214/) (2023)
- [Learning Hamiltonian Systems with Mono-Implicit Runge-Kutta Methods](https://mlsanity.com/paper/2303.03769/) (2023)
- [Numerical and statistical analysis of NeuralODE with Runge-Kutta time integration](https://mlsanity.com/paper/2503.10729/) (2025)
- [Diffusion Secant Alignment for Score-Based Density Ratio Estimation](https://mlsanity.com/paper/2509.04852/) (2025)
- [Analysis of Numerical Integration in RNN-Based Residuals for Fault Diagnosis of Dynamic Systems](https://mlsanity.com/paper/2305.04670/) (2023)
- [Sharpen Your Flow: Sharpness-Aware Sampling for Flow Matching](https://mlsanity.com/paper/2605.11547/) (2026)
- [Pseudo-Hamiltonian system identification](https://mlsanity.com/paper/2305.06920/) (2023)
- [A Unified Framework for Neural Computation and Learning Over Time](https://mlsanity.com/paper/2409.12038/) (2024)
- [Artifacts of Numerical Integration in Learning Dynamical Systems](https://mlsanity.com/paper/2507.14491/) (2025)
- [FNODE: Flow-Matching for data-driven simulation of constrained multibody systems](https://mlsanity.com/paper/2509.00183/) (2025)
- [Controlled oscillation modeling using port-Hamiltonian neural networks](https://mlsanity.com/paper/2602.15704/) (2026)
- [Time-vectorized numerical integration for systems of ODEs](https://mlsanity.com/paper/2310.08649/) (2023)
- [Federated Spectral Graph Transformers Meet Neural Ordinary Differential Equations for Non-IID Graphs](https://mlsanity.com/paper/2504.11808/) (2025)
- [Interpretability and Generalization Bounds for Learning Spatial Physics](https://mlsanity.com/paper/2506.15199/) (2025)
- [RTS Smoother-Guided Learning of Physics-Based Neural Differential Models](https://mlsanity.com/paper/2607.15180/) (2026)
- [Variational Streaming Flow: Probabilistic Forecasting in Physical Time](https://mlsanity.com/paper/2610.00976/) (2026)

## Abstract (arXiv)

The combination of ordinary differential equations and neural networks, i.e., neural ordinary differential equations (Neural ODE), has been widely studied from various angles. However, deciphering the numerical integration in Neural ODE is still an open challenge, as many researches demonstrated that numerical integration significantly affects the performance of the model. In this paper, we propose the inverse modified differential equations (IMDE) to clarify the influence of numerical integration on training Neural ODE models. IMDE is determined by the learning task and the employed ODE solver. It is shown that training a Neural ODE model actually returns a close approximation of the IMDE, rather than the true ODE. With the help of IMDE, we deduce that (i) the discrepancy between the learned model and the true ODE is bounded by the sum of discretization error and learning loss; (ii) Neural ODE using non-symplectic numerical integration fail to learn conservation laws theoretically. Several experiments are performed to numerically verify our theoretical analysis.