# MomentumRNN: Integrating Momentum into Recurrent Neural Networks

arXiv: https://arxiv.org/abs/2006.06919 · HTML: https://mlsanity.com/paper/2006.06919/

## Key facts

- Cited by 18 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 12 Jun 2020.

## Built on by (most cited first, 18 of 18)

- [Machine learning accelerated computational fluid dynamics](https://mlsanity.com/paper/2102.01010/) (2021)
- [Heavy Ball Neural Ordinary Differential Equations](https://mlsanity.com/paper/2110.04840/) (2021)
- [Momentum Residual Neural Networks](https://mlsanity.com/paper/2102.07870/) (2021)
- [Improving Transformers with Probabilistic Attention Keys](https://mlsanity.com/paper/2110.08678/) (2021)
- [How Does Momentum Benefit Deep Neural Networks Architecture Design? A Few Case Studies](https://mlsanity.com/paper/2110.07034/) (2021)
- [Adaptive and Implicit Regularization for Matrix Completion](https://mlsanity.com/paper/2208.05640/) (2022)
- [CAMEx: Curvature-aware Merging of Experts](https://mlsanity.com/paper/2502.18821/) (2025)
- [Momentum Transformer: Closing the Performance Gap Between Self-attention and Its Linearization](https://mlsanity.com/paper/2208.00579/) (2022)
- [Diffusion Sampling with Momentum for Mitigating Divergence Artifacts](https://mlsanity.com/paper/2307.11118/) (2023)
- [Expert Merging in Sparse Mixture of Experts with Nash Bargaining](https://mlsanity.com/paper/2510.16138/) (2025)
- [MuonSSM: Orthogonalizing State Space Models for Sequence Modeling](https://mlsanity.com/paper/2606.30461/) (2026)
- [Learning POD of Complex Dynamics Using Heavy-ball Neural ODEs](https://mlsanity.com/paper/2202.12373/) (2022)
- [Demystifying the Token Dynamics of Deep Selective State Space Models](https://mlsanity.com/paper/2410.03292/) (2024)
- [Second Order Drifting Models](https://mlsanity.com/paper/2608.07924/) (2026)
- [AIR-Net: Adaptive and Implicit Regularization Neural Network for Matrix Completion](https://mlsanity.com/paper/2110.07557/) (2021)
- [DenseBAM-GI: Attention Augmented DeneseNet with momentum aided GRU for HMER](https://mlsanity.com/paper/2306.16482/) (2023)
- [MMA: A Momentum Mamba Architecture for Human Activity Recognition with Inertial Sensors](https://mlsanity.com/paper/2511.21550/) (2025)
- [Leakage and Second-Order Dynamics Improve Hippocampal RNN Replay](https://mlsanity.com/paper/2602.18401/) (2026)

## Abstract (arXiv)

Designing deep neural networks is an art that often involves an expensive search over candidate architectures. To overcome this for recurrent neural nets (RNNs), we establish a connection between the hidden state dynamics in an RNN and gradient descent (GD). We then integrate momentum into this framework and propose a new family of RNNs, called em MomentumRNNs. We theoretically prove and numerically demonstrate that MomentumRNNs alleviate the vanishing gradient issue in training RNNs. We study the momentum long-short term memory (MomentumLSTM) and verify its advantages in convergence speed and accuracy over its LSTM counterpart across a variety of benchmarks. We also demonstrate that MomentumRNN is applicable to many types of recurrent cells, including those in the state-of-the-art orthogonal RNNs. Finally, we show that other advanced momentum-based optimization methods, such as Adam and Nesterov accelerated gradients with a restart, can be easily incorporated into the MomentumRNN framework for designing new recurrent cells with even better performance. The code is available at https://github.com/minhtannguyen/MomentumRNN.