The Vanishing Gradient Problem for Stiff Neural Differential Equations
Not on the map. Why?
Abstract (arXiv)
Gradient-based optimization of neural differential equations and other parameterized dynamical systems fundamentally relies on the ability to differentiate numerical solutions with respect to model parameters. In stiff systems, it has been observed that sensitivities to parameters controlling fast-decaying modes become vanishingly small during training, leading to optimization difficulties. In this paper, we show that this vanishing gradient phenomenon is not an artifact of any particular method, but a universal feature of all A-stable and L-stable stiff numerical integration schemes. We analyze the rational stability function for general stiff integration schemes and demonstrate that the relevant parameter sensitivities, governed by the derivative of the stability function, decay to zero for large stiffness. Explicit formulas for common stiff integration schemes are provided, which illustrate the mechanism in detail. Finally, we rigorously prove that the slowest possible rate of decay for the derivative of the stability function is O(|z|-1), revealing a fundamental limitation: all A-stable time-stepping methods inevitably suppress parameter gradients in stiff regimes, posing a significant barrier for training and parameter identification in stiff neural ODEs.
Key facts. Cited by 3 papers in the map (data through 14 Oct 2026). Builds on 59 of its 103 references that are in the map. First posted on arXiv 2 Aug 2025.
Builds on
59 of 103 references.
References · 59
- 2017Attention Is All You Needcited by 60,361
- 2015Deep Residual Learning for Image Recognitioncited by 52,096
- 2015Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shiftcited by 7,005
- 2016Layer Normalizationcited by 5,669
- 2014Learning Phrase Representations using RNN Encoder-Decoder for Statistical Machine Translationcited by 4,057
- 2018Neural Ordinary Differential Equationscited by 3,779
- 2015Delving Deep into Rectifiers: Surpassing Human-Level Performance on ImageNet Classificationcited by 3,611
- 2016Gaussian Error Linear Units (GELUs)cited by 3,502
- 2016Wide Residual Networkscited by 3,070
- 2014Empirical Evaluation of Gated Recurrent Neural Networks on Sequence Modelingcited by 2,367
- 2016Identity Mappings in Deep Residual Networkscited by 1,965
- 2018Group Normalizationcited by 1,524
38 more
- 2014On the Properties of Neural Machine Translation: Encoder-Decoder Approachescited by 1,304
- 2015Fast and Accurate Deep Network Learning by Exponential Linear Units (ELUs)cited by 1,070
- 2020Learning Mesh-Based Simulation with Graph Networkscited by 868
- 2015Automatic differentiation in machine learning: a surveycited by 855
- 2013Exact solutions to the nonlinear dynamics of learning in deep linear neural networkscited by 763
- 2013Speech Recognition with Deep Recurrent Neural Networkscited by 669
- 2022Scientific Machine Learning through Physics-Informed Neural Networks: Where we are and What's nextcited by 608
- 2020Neural Controlled Differential Equations for Irregular Time Seriescited by 506
- 2021Physics-informed neural networks (PINNs) for fluid mechanics: A reviewcited by 473
- 2021A Survey of Transformerscited by 401
- 2014On the Number of Linear Regions of Deep Neural Networkscited by 381
- 2018How Does Batch Normalization Help Optimization?cited by 357
- 2020Universal Differential Equations for Scientific Machine Learningcited by 353
- 2016On the Expressive Power of Deep Neural Networkscited by 306
- 2015Unitary Evolution Recurrent Neural Networkscited by 294
- 2020Scalable Gradients for Stochastic Differential Equationscited by 271
- 2022On Neural Differential Equationscited by 268
- 2016Exponential expressivity in deep neural networks through transient chaoscited by 261
- 2015Highway Networkscited by 260
- 2020How to train your neural ODE: the world of Jacobian and kinetic regularizationcited by 228
- 2017Hidden Physics Models: Machine Learning of Nonlinear Partial Differential Equationscited by 208
- 2017Physics Informed Deep Learning (Part II): Data-driven Discovery of Nonlinear Partial Differential Equationscited by 161
- 2019Neural Jump Stochastic Differential Equationscited by 150
- 2020OT-Flow: Fast and Accurate Continuous Normalizing Flows via Optimal Transportcited by 143
- 2020Neural Rough Differential Equations for Long Time Seriescited by 143
- 2015All you need is a good initcited by 124
- 2018How to Start Training: The Effect of Initialization and Architecturecited by 109
- 2019ANODE: Unconditionally Accurate Memory-Efficient Gradients for Neural ODEscited by 84
- 2020Learning Differential Equations that are Easy to Solvecited by 81
- 2021Stiff Neural Ordinary Differential Equationscited by 78
- 2017Numerical Gaussian Processes for Time-dependent and Non-linear Partial Differential Equationscited by 72
- 2020STEER: Simple Temporal Regularization For Neural ODEscited by 58
- 2022Stabilized Neural Ordinary Differential Equations for Long-Time Forecasting of Dynamical Systemscited by 50
- 2020Bayesian Neural Ordinary Differential Equationscited by 45
- 2020Discretize-Optimize vs. Optimize-Discretize for Time-Series Regression and Continuous Normalizing Flowscited by 37
- 2020ChemNODE: A Neural Ordinary Differential Equations Approach for Chemical Kinetics Solverscited by 26
- 2021Opening the Blackbox: Accelerating Neural Differential Equations by Regularizing Internal Solver Heuristicscited by 26
- 2020Stable Neural Flowscited by 25
50 most cited of 59 All 59 (JSON)
Built on by
Citing papers per year: 2025: 1; 2026: 2.
3 papers.