mlsanityData to 7 Oct 2026

Heavy Ball Neural Ordinary Differential Equations

arXiv 2110.04840 ↗First posted 10 Oct 2021cs.LG cs.AI cs.NAcited by 42

Not on the map. Why?

Abstract (arXiv)

We propose heavy ball neural ordinary differential equations (HBNODEs), leveraging the continuous limit of the classical momentum accelerated gradient descent, to improve neural ODEs (NODEs) training and inference. HBNODEs have two properties that imply practical advantages over NODEs: (i) The adjoint state of an HBNODE also satisfies an HBNODE, accelerating both forward and backward ODE solvers, thus significantly reducing the number of function evaluations (NFEs) and improving the utility of the trained models. (ii) The spectrum of HBNODEs is well structured, enabling effective learning of long-term dependencies from complex sequential data. We verify the advantages of HBNODEs over NODEs on benchmark tasks, including image classification, learning complex dynamics, and sequential modeling. Our method requires remarkably fewer forward and backward NFEs, is more accurate, and learns long-term dependencies more effectively than the other ODE-based neural network models. Code is available at urlhttps://github.com/hedixia/HeavyBallNODE.

Key facts. Cited by 42 papers in the map (data through 7 Oct 2026). Builds on 48 of its 75 references that are in the map. First posted on arXiv 10 Oct 2021.

Builds on

48 of 75 references.

References · 48

  1. 2017Attention Is All You Needcited by 59,932
  2. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  3. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  4. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  5. 2014Learning Phrase Representations using RNN Encoder-Decoder for Statistical Machine Translationcited by 4,016
  6. 2018Neural Ordinary Differential Equationscited by 3,733
  7. 2016Identity Mappings in Deep Residual Networkscited by 1,960
  8. 2020Neural Controlled Differential Equations for Irregular Time Seriescited by 501
  9. 2019Augmented Neural ODEscited by 396
  10. 2014Stochastic Gradient Hamiltonian Monte Carlocited by 341
  11. 2017Stable Architectures for Deep Neural Networkscited by 340
  12. 2015Unitary Evolution Recurrent Neural Networkscited by 291
36 more
  1. 2020Scalable Gradients for Stochastic Differential Equationscited by 264
  2. 2020How to train your neural ODE: the world of Jacobian and kinetic regularizationcited by 228
  3. 2015A Differential Equation for Modeling Nesterov's Accelerated Gradient Method: Theory and Insightscited by 203
  4. 2017Beyond Finite Layer Neural Networks: Bridging Deep Architectures and Numerical Differential Equationscited by 203
  5. 2019Symplectic ODE-Net: Learning Hamiltonian Dynamics with Controlcited by 183
  6. 2019Neural Stochastic Differential Equations: Deep Latent Gaussian Models in the Diffusion Limitcited by 168
  7. 2019Graph Neural Ordinary Differential Equationscited by 153
  8. 2020Dissecting Neural ODEscited by 150
  9. 2019Neural Jump Stochastic Differential Equationscited by 147
  10. 2016Full-Capacity Unitary Recurrent Neural Networkscited by 90
  11. 2020Lipschitz Recurrent Neural Networkscited by 88
  12. 2017On orthogonality and learning recurrent networks with long term dependenciescited by 82
  13. 2017Orthogonal Recurrent Neural Networks with Scaled Cayley Transformcited by 82
  14. 2020Learning Long-Term Dependencies in Irregularly-Sampled Time Seriescited by 82
  15. 2020Learning Differential Equations that are Easy to Solvecited by 81
  16. 2019Deep Learning via Dynamical Systems: An Approximation Perspectivecited by 78
  17. 2018Understanding the Acceleration Phenomenon via High-Resolution Differential Equationscited by 68
  18. 2020On Second Order Behaviour in Augmented Neural ODEscited by 68
  19. 2016Tunable Efficient Unitary Neural Networks (EUNN) and their application to RNNscited by 62
  20. 2020Learning Continuous System Dynamics from Irregularly-Sampled Partial Observationscited by 60
  21. 2020STEER: Simple Temporal Regularization For Neural ODEscited by 58
  22. 2021Neural ODE Processescited by 57
  23. 2016Efficient Orthogonal Parametrisation of Recurrent Neural Networks Using Householder Reflectionscited by 56
  24. 2021MALI: A memory efficient and reverse accurate integrator for Neural ODEscited by 42
  25. 2019ODE2VAE: Deep generative second order ODEs with Bayesian neural networkscited by 36
  26. 2021Momentum Residual Neural Networkscited by 36
  27. 2019Continuous Time Analysis of Momentum Methodscited by 32
  28. 2020Model-based Reinforcement Learning for Semi-Markov Decision Processes with Neural ODEscited by 32
  29. 2016A Lyapunov Analysis of Momentum Methods in Optimizationcited by 28
  30. 2019Acceleration via Symplectic Discretization of High-Resolution Differential Equationscited by 27
  31. 2019SNODE: Spectral Discretization of Neural ODEs for System Identificationcited by 26
  32. 2021Opening the Blackbox: Accelerating Neural Differential Equations by Regularizing Internal Solver Heuristicscited by 26
  33. 2020Scheduled Restart Momentum for Accelerated Stochastic Gradient Descentcited by 24
  34. 2020MomentumRNN: Integrating Momentum into Recurrent Neural Networkscited by 18
  35. 2018Optimization Algorithm Inspired Deep Neural Network Structure Designcited by 14
  36. 2017Understanding the Learned Iterative Soft Thresholding Algorithm with matrix factorizationcited by 9

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 3; 2022: 13; 2023: 11; 2024: 6; 2025: 7; 2026: 2.

42 papers.

Sort:

By area:

  1. 2021Improving Transformers with Probabilistic Attention Keyscited by 28
  2. 2022On Numerical Integration in Neural Ordinary Differential Equationscited by 21
  3. 2025ODE-GS: Latent ODEs for Dynamic Scene Extrapolation with 3D Gaussian Splattingcited by 11
  4. 2021How Does Momentum Benefit Deep Neural Networks Architecture Design? A Few Case Studiescited by 10
  5. 2022torchode: A Parallel ODE Solver for PyTorchcited by 10
  6. 2023Modulated Neural ODEscited by 10
  7. 2024MomentumSMoE: Integrating Momentum into Sparse Mixture of Expertscited by 9
  8. 2024Learning Physics Informed Neural ODEs With Partial Measurementscited by 9
  9. 2022TimeKit: A Time-series Forecasting-based Upgrade Kit for Collaborative Filteringcited by 7
  10. 2023PGODE: Towards High-quality System Dynamics Modelingcited by 7
  11. 2022Proximal Implicit ODE Solvers for Accelerating Learning Neural ODEscited by 6
  12. 2022Continuous Deep Equilibrium Models: Training Neural ODEs faster by integrating them to Infinitycited by 5
30 more
  1. 2022Adaptive and Implicit Regularization for Matrix Completioncited by 5
  2. 2022Gated Recurrent Neural Networks with Weighted Time-Delay Feedbackcited by 5
  3. 2022Revisiting the acceleration phenomenon via high-resolution differential equationscited by 5
  4. 2023Stability-Informed Initialization of Neural Ordinary Differential Equationscited by 5
  5. 2022Momentum Transformer: Closing the Performance Gap Between Self-attention and Its Linearizationcited by 4
  6. 2023Locally Regularized Neural Differential Equations: Some Black Boxes Were Meant to Remain Closed!cited by 4
  7. 2025Structured Kolmogorov-Arnold Neural ODEs for Interpretable Learning and Symbolic Discovery of Nonlinear Dynamicscited by 4
  8. 2022AdamNODEs: When Neural ODE Meets Adaptive Moment Estimationcited by 3
  9. 2023Learning PDE Solution Operator for Continuous Modeling of Time-Seriescited by 3
  10. 2023Diffusion Sampling with Momentum for Mitigating Divergence Artifactscited by 3
  11. 2023Faster Training of Neural ODEs Using Gauss-Legendre Quadraturecited by 3
  12. 2022Learning POD of Complex Dynamics Using Heavy-ball Neural ODEscited by 2
  13. 2023Trainability, Expressivity and Interpretability in Gated Neural ODEscited by 2
  14. 2024The Disappearance of Timestep Embedding in Modern Time-Dependent Neural Networkscited by 2
  15. 2025Efficient Manifold-Constrained Neural ODE for High-Dimensional Datasetscited by 2
  16. 2025Deep Neural Networks Inspired by Differential Equationscited by 2
  17. 2025Expert Merging in Sparse Mixture of Experts with Nash Bargainingcited by 2
  18. 2023Linear convergence of forward-backward accelerated algorithms without knowledge of the modulus of strong convexitycited by 1
  19. 2023On Tuning Neural ODE for Stability, Consistency and Faster Convergencecited by 1
  20. 2024Demystifying the Token Dynamics of Deep Selective State Space Modelscited by 1
  21. 2024Discover physical concepts and equations with machine learningcited by 1
  22. 2026Second Order Drifting Modelscited by 1
  23. 2021AIR-Net: Adaptive and Implicit Regularization Neural Network for Matrix Completioncited by 0
  24. 2022A dynamical systems based framework for dimension reductioncited by 0
  25. 2022Improved Batching Strategy For Irregular Time-Series ODEcited by 0
  26. 2023Operator-learning-inspired Modeling of Neural Ordinary Differential Equationscited by 0
  27. 2024Neural Controlled Differential Equations with Quantum Hidden Evolutionscited by 0
  28. 2025Modeling Neural Networks with Privacy Using Neural Stochastic Differential Equationscited by 0
  29. 2025Modeling Time Series Dynamics with Fourier Ordinary Differential Equationscited by 0
  30. 2026Semantic Privacy Protection with Utility Preservation for 3D Point Cloudscited by 0