# Stiff Neural Ordinary Differential Equations

arXiv: https://arxiv.org/abs/2103.15341 · HTML: https://mlsanity.com/paper/2103.15341/

## Key facts

- Cited by 78 papers in the map (data through 14 Oct 2026).
- Builds on 14 of its 43 references that are in the map.
- First posted on arXiv 29 Mar 2021.

## Builds on (most cited first, 14 of 14)

- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift](https://mlsanity.com/paper/1502.03167/) (2015)
- [Neural Ordinary Differential Equations](https://mlsanity.com/paper/1806.07366/) (2018)
- [Gaussian Error Linear Units (GELUs)](https://mlsanity.com/paper/1606.08415/) (2016)
- [Deep Equilibrium Models](https://mlsanity.com/paper/1909.01377/) (2019)
- [OptNet: Differentiable Optimization as a Layer in Neural Networks](https://mlsanity.com/paper/1703.00443/) (2017)
- [Universal Differential Equations for Scientific Machine Learning](https://mlsanity.com/paper/2001.04385/) (2020)
- [Understanding and mitigating gradient pathologies in physics-informed neural networks](https://mlsanity.com/paper/2001.04536/) (2020)
- [ANODE: Unconditionally Accurate Memory-Efficient Gradients for Neural ODEs](https://mlsanity.com/paper/1902.10298/) (2019)
- [STEER: Simple Temporal Regularization For Neural ODEs](https://mlsanity.com/paper/2006.10711/) (2020)
- [DiffEqFlux.jl - A Julia Library for Neural Differential Equations](https://mlsanity.com/paper/1902.02376/) (2019)
- [Autonomous Discovery of Unknown Reaction Pathways from Data by Chemical Reaction Neural Network](https://mlsanity.com/paper/2002.09062/) (2020)
- [Universal Battery Performance and Degradation Model for Electric Aircraft](https://mlsanity.com/paper/2008.01527/) (2020)
- [Accelerating Simulation of Stiff Nonlinear Systems using Continuous-Time Echo State Networks](https://mlsanity.com/paper/2010.04004/) (2020)

## Built on by (most cited first, 30 of 78)

- [KAN-ODEs: Kolmogorov-Arnold Network Ordinary Differential Equations for Learning Dynamical Systems and Hidden Physics](https://mlsanity.com/paper/2407.04192/) (2024)
- [Effectively Modeling Time Series with Simple Discrete State Spaces](https://mlsanity.com/paper/2303.09489/) (2023)
- [Physics-Constrained Flow Matching: Sampling Generative Models with Hard Constraints](https://mlsanity.com/paper/2506.04171/) (2025)
- [Neural Controlled Differential Equations for Online Prediction Tasks](https://mlsanity.com/paper/2106.11028/) (2021)
- [Deep Latent State Space Models for Time-Series Generation](https://mlsanity.com/paper/2212.12749/) (2022)
- [Learning stiff chemical kinetics using extended deep neural operators](https://mlsanity.com/paper/2302.12645/) (2023)
- [DAE-PINN: A Physics-Informed Neural Network Model for Simulating Differential-Algebraic Equations with Application to Power Networks](https://mlsanity.com/paper/2109.04304/) (2021)
- [Neural Laplace: Learning diverse classes of differential equations in the Laplace domain](https://mlsanity.com/paper/2206.04843/) (2022)
- [Interpretable Polynomial Neural Ordinary Differential Equations](https://mlsanity.com/paper/2208.05072/) (2022)
- [Differentiable Programming for Earth System Modeling](https://mlsanity.com/paper/2208.13825/) (2022)
- [Comprehensive Review of Neural Differential Equations for Time Series Analysis](https://mlsanity.com/paper/2502.09885/) (2025)
- [DySMHO: Data-Driven Discovery of Governing Equations for Dynamical Systems via Moving Horizon Optimization](https://mlsanity.com/paper/2108.00069/) (2021)
- [Parsimonious Physics-Informed Random Projection Neural Networks for Initial-Value Problems of ODEs and index-1 DAEs](https://mlsanity.com/paper/2203.05337/) (2022)
- [Stabilized Neural Differential Equations for Learning Dynamics with Explicit Constraints](https://mlsanity.com/paper/2306.09739/) (2023)
- [When are dynamical systems learned from time series data statistically accurate?](https://mlsanity.com/paper/2411.06311/) (2024)
- [Residual-based physics-informed transfer learning: A hybrid method for accelerating long-term CFD simulations via deep learning](https://mlsanity.com/paper/2206.06817/) (2022)
- [Homotopy-based training of NeuralODEs for accurate dynamics discovery](https://mlsanity.com/paper/2210.01407/) (2022)
- [Finite volume method network for acceleration of unsteady computational fluid dynamics: non-reacting and reacting flows](https://mlsanity.com/paper/2105.03332/) (2021)
- [On the Forward Invariance of Neural ODEs](https://mlsanity.com/paper/2210.04763/) (2022)
- [Training Stiff Neural Ordinary Differential Equations with Implicit Single-Step Methods](https://mlsanity.com/paper/2410.05592/) (2024)
- [From Fourier to Neural ODEs: Flow Matching for Modeling Complex Systems](https://mlsanity.com/paper/2405.11542/) (2024)
- [DUE: A Deep Learning Framework and Library for Modeling Unknown Equations](https://mlsanity.com/paper/2504.10373/) (2025)
- [AdjointDEIS: Efficient Gradients for Diffusion Models](https://mlsanity.com/paper/2405.15020/) (2024)
- [Learning Hamiltonian neural Koopman operator and simultaneously sustaining and discovering conservation law](https://mlsanity.com/paper/2406.02154/) (2024)
- [Learning Energy Conserving Dynamics Efficiently with Hamiltonian Gaussian Processes](https://mlsanity.com/paper/2303.01925/) (2023)
- [Bayesian polynomial neural networks and polynomial neural ordinary differential equations](https://mlsanity.com/paper/2308.10892/) (2023)
- [Learning Neural Differential Algebraic Equations via Operator Splitting](https://mlsanity.com/paper/2403.12938/) (2024)
- [Mitigating mode collapse in normalizing flows by annealing with an adaptive schedule: Application to parameter estimation](https://mlsanity.com/paper/2505.03652/) (2025)
- [Training Stiff Neural Ordinary Differential Equations with Explicit Exponential Integration Methods](https://mlsanity.com/paper/2412.01181/) (2024)
- [Locally Regularized Neural Differential Equations: Some Black Boxes Were Meant to Remain Closed!](https://mlsanity.com/paper/2303.02262/) (2023)

## Abstract (arXiv)

Neural Ordinary Differential Equations (ODE) are a promising approach to learn dynamic models from time-series data in science and engineering applications. This work aims at learning Neural ODE for stiff systems, which are usually raised from chemical kinetic modeling in chemical and biological systems. We first show the challenges of learning neural ODE in the classical stiff ODE systems of Robertson's problem and propose techniques to mitigate the challenges associated with scale separations in stiff systems. We then present successful demonstrations in stiff systems of Robertson's problem and an air pollution problem. The demonstrations show that the usage of deep networks with rectified activations, proper scaling of the network outputs as well as loss functions, and stabilized gradient calculations are the key techniques enabling the learning of stiff neural ODE. The success of learning stiff neural ODE opens up possibilities of using neural ODEs in applications with widely varying time-scales, like chemical dynamics in energy conversion, environmental engineering, and the life sciences.