# Stability-Informed Initialization of Neural Ordinary Differential Equations

arXiv: https://arxiv.org/abs/2311.15890 · HTML: https://mlsanity.com/paper/2311.15890/

## Key facts

- Cited by 5 papers in the map (data through 14 Oct 2026).
- Builds on 40 of its 67 references that are in the map.
- First posted on arXiv 27 Nov 2023.

## Builds on (most cited first, 30 of 40)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [PyTorch: An Imperative Style, High-Performance Deep Learning Library](https://mlsanity.com/paper/1912.01703/) (2019)
- [U-Net: Convolutional Networks for Biomedical Image Segmentation](https://mlsanity.com/paper/1505.04597/) (2015)
- [Flow Matching for Generative Modeling](https://mlsanity.com/paper/2210.02747/) (2022)
- [Auto-Encoding Variational Bayes](https://mlsanity.com/paper/1312.6114/) (2013)
- [Neural Ordinary Differential Equations](https://mlsanity.com/paper/1806.07366/) (2018)
- [Delving Deep into Rectifiers: Surpassing Human-Level Performance on ImageNet Classification](https://mlsanity.com/paper/1502.01852/) (2015)
- [Efficiently Modeling Long Sequences with Structured State Spaces](https://mlsanity.com/paper/2111.00396/) (2021)
- [Fourier Features Let Networks Learn High Frequency Functions in Low Dimensional Domains](https://mlsanity.com/paper/2006.10739/) (2020)
- [WaveNet: A Generative Model for Raw Audio](https://mlsanity.com/paper/1609.03499/) (2016)
- [An Empirical Evaluation of Generic Convolutional and Recurrent Networks for Sequence Modeling](https://mlsanity.com/paper/1803.01271/) (2018)
- [On the Spectral Bias of Neural Networks](https://mlsanity.com/paper/1806.08734/) (2018)
- [Fast and Accurate Deep Network Learning by Exponential Linear Units (ELUs)](https://mlsanity.com/paper/1511.07289/) (2015)
- [Exact solutions to the nonlinear dynamics of learning in deep linear neural networks](https://mlsanity.com/paper/1312.6120/) (2013)
- [An Empirical Investigation of Catastrophic Forgetting in Gradient-Based Neural Networks](https://mlsanity.com/paper/1312.6211/) (2013)
- [HiPPO: Recurrent Memory with Optimal Polynomial Projections](https://mlsanity.com/paper/2008.07669/) (2020)
- [BranchyNet: Fast Inference via Early Exiting from Deep Neural Networks](https://mlsanity.com/paper/1709.01686/) (2017)
- [Neural Controlled Differential Equations for Irregular Time Series](https://mlsanity.com/paper/2005.08926/) (2020)
- [How to train your neural ODE: the world of Jacobian and kinetic regularization](https://mlsanity.com/paper/2002.02798/) (2020)
- [It's Raw! Audio Generation with State-Space Models](https://mlsanity.com/paper/2202.09729/) (2022)
- [Dissecting Neural ODEs](https://mlsanity.com/paper/2002.08071/) (2020)
- [Frequency Bias in Neural Networks for Input of Non-Uniform Density](https://mlsanity.com/paper/2003.04560/) (2020)
- [Hamiltonian Generative Networks](https://mlsanity.com/paper/1909.13789/) (2019)
- [A Simple Way to Initialize Recurrent Networks of Rectified Linear Units](https://mlsanity.com/paper/1504.00941/) (2015)
- [ANODE: Unconditionally Accurate Memory-Efficient Gradients for Neural ODEs](https://mlsanity.com/paper/1902.10298/) (2019)
- [On Robustness of Neural Ordinary Differential Equations](https://mlsanity.com/paper/1910.05513/) (2019)
- [Adaptive Checkpoint Adjoint Method for Gradient Estimation in Neural ODE](https://mlsanity.com/paper/2006.02493/) (2020)
- [Learning Continuous System Dynamics from Irregularly-Sampled Partial Observations](https://mlsanity.com/paper/2011.03880/) (2020)
- [Approximation Capabilities of Neural ODEs and Invertible Residual Networks](https://mlsanity.com/paper/1907.12998/) (2019)
- [Heavy Ball Neural Ordinary Differential Equations](https://mlsanity.com/paper/2110.04840/) (2021)

## Built on by (most cited first, 5 of 5)

- [Lyapunov Probes for Hallucination Detection in Large Foundation Models](https://mlsanity.com/paper/2603.06081/) (2026)
- [Learning the Simplest Neural ODE](https://mlsanity.com/paper/2505.02019/) (2025)
- [Jacobian Regularization Stabilizes Long-Term Integration of Neural Differential Equations](https://mlsanity.com/paper/2602.04608/) (2026)
- [A neural operator for predicting vibration frequency response curves from limited data](https://mlsanity.com/paper/2603.10149/) (2026)
- [Latent Laplace Diffusion for Irregular Multivariate Time Series](https://mlsanity.com/paper/2605.19805/) (2026)

## Abstract (arXiv)

This paper addresses the training of Neural Ordinary Differential Equations (neural ODEs), and in particular explores the interplay between numerical integration techniques, stability regions, step size, and initialization techniques. It is shown how the choice of integration technique implicitly regularizes the learned model, and how the solver's corresponding stability region affects training and prediction performance. From this analysis, a stability-informed parameter initialization technique is introduced. The effectiveness of the initialization method is displayed across several learning benchmarks and industrial applications.