mlsanityData to 7 Oct 2026

How Does Momentum Benefit Deep Neural Networks Architecture Design? A Few Case Studies

arXiv 2110.07034 ↗First posted 13 Oct 2021cs.LG cs.NA math.DScited by 10

Not on the map. Why?

Abstract (arXiv)

We present and review an algorithmic and theoretical framework for improving neural network architecture design via momentum. As case studies, we consider how momentum can improve the architecture design for recurrent neural networks (RNNs), neural ordinary differential equations (ODEs), and transformers. We show that integrating momentum into neural network architectures has several remarkable theoretical and empirical benefits, including 1) integrating momentum into RNNs and neural ODEs can overcome the vanishing gradient issues in training RNNs and neural ODEs, resulting in effective learning long-term dependencies. 2) momentum in neural ODEs can reduce the stiffness of the ODE dynamics, which significantly enhances the computational efficiency in training and testing. 3) momentum can improve the efficiency and accuracy of transformers.

Key facts. Cited by 10 papers in the map (data through 7 Oct 2026). Builds on 83 of its 121 references that are in the map. First posted on arXiv 13 Oct 2021.

Builds on

83 of 121 references.

References · 83

  1. 2017Attention Is All You Needcited by 59,932
  2. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  3. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  4. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  5. 2020Language Models are Few-Shot Learnerscited by 32,199
  6. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  7. 2017Towards Deep Learning Models Resistant to Adversarial Attackscited by 6,326
  8. 2013Intriguing properties of neural networkscited by 5,114
  9. 2020Training data-efficient image transformers & distillation through attentioncited by 4,560
  10. 2016SQuAD: 100,000+ Questions for Machine Comprehension of Textcited by 4,115
  11. 2014Learning Phrase Representations using RNN Encoder-Decoder for Statistical Machine Translationcited by 4,016
  12. 2018Neural Ordinary Differential Equationscited by 3,733
38 more
  1. 2020Longformer: The Long-Document Transformercited by 3,362
  2. 2015Neural Machine Translation of Rare Words with Subword Unitscited by 2,898
  3. 2017A Broad-Coverage Challenge Corpus for Sentence Understanding through Inferencecited by 2,665
  4. 2020Transformers are RNNs: Fast Autoregressive Transformers with Linear Attentioncited by 1,979
  5. 2016Identity Mappings in Deep Residual Networkscited by 1,960
  6. 2019Transformer-XL: Attentive Language Models Beyond a Fixed-Length Contextcited by 1,830
  7. 2020Big Bird: Transformers for Longer Sequencescited by 1,673
  8. 2020Reformer: The Efficient Transformercited by 1,658
  9. 2020Rethinking Attention with Performerscited by 1,584
  10. 2019Generating Long Sequences with Sparse Transformerscited by 1,523
  11. 2020Linformer: Self-Attention with Linear Complexitycited by 1,462
  12. 2020Efficient Transformers: A Surveycited by 784
  13. 2018Image Transformercited by 770
  14. 2020Long Range Arena: A Benchmark for Efficient Transformerscited by 696
  15. 2020Neural Controlled Differential Equations for Irregular Time Seriescited by 501
  16. 2020Efficient Content-Based Sparse Attention with Routing Transformerscited by 469
  17. 2021Nystr"omformer: A Nystr"om-Based Algorithm for Approximating Self-Attentioncited by 468
  18. 2017PixelCNN++: Improving the PixelCNN with Discretized Logistic Mixture Likelihood and Other Modificationscited by 415
  19. 2019Augmented Neural ODEscited by 396
  20. 2018Efficient Attention: Attention with Linear Complexitiescited by 366
  21. 2019Axial Attention in Multidimensional Transformerscited by 350
  22. 2021Random Feature Attentioncited by 350
  23. 2017Stable Architectures for Deep Neural Networkscited by 340
  24. 2018Generating Wikipedia by Summarizing Long Sequencescited by 333
  25. 2015Unitary Evolution Recurrent Neural Networkscited by 291
  26. 2018Adversarial Attacks and Defences: A Surveycited by 259
  27. 2020Sparse Sinkhorn Attentioncited by 250
  28. 2020How to train your neural ODE: the world of Jacobian and kinetic regularizationcited by 228
  29. 2018Music Transformercited by 207
  30. 2020Synthesizer: Rethinking Self-Attention in Transformer Modelscited by 204
  31. 2015A Differential Equation for Modeling Nesterov's Accelerated Gradient Method: Theory and Insightscited by 203
  32. 2018Deterministic Non-Autoregressive Neural Sequence Modeling by Iterative Refinementcited by 194
  33. 2019The Evolved Transformercited by 180
  34. 2019Blockwise Self-Attention for Long Document Understandingcited by 158
  35. 2020Dissecting Neural ODEscited by 150
  36. 2018Character-Level Language Modeling with Deeper Self-Attentioncited by 138
  37. 2019Cheap Orthogonal Constraints in Neural Networks: A Simple Parametrization of the Orthogonal and Unitary Groupcited by 133
  38. 2015A Simple Way to Initialize Recurrent Networks of Rectified Linear Unitscited by 130

50 most cited of 83 All 83 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 1; 2022: 2; 2023: 1; 2024: 2; 2025: 2; 2026: 2.

10 papers.

  1. 2021Improving Transformers with Probabilistic Attention Keyscited by 28
  2. 2024MomentumSMoE: Integrating Momentum into Sparse Mixture of Expertscited by 9
  3. 2022Momentum Transformer: Closing the Performance Gap Between Self-attention and Its Linearizationcited by 4
  4. 2025CAMEx: Curvature-aware Merging of Expertscited by 4
  5. 2023Physics-informed Machine Learning for Calibrating Macroscopic Traffic Flow Modelscited by 3
  6. 2022Learning POD of Complex Dynamics Using Heavy-ball Neural ODEscited by 2
  7. 2025Expert Merging in Sparse Mixture of Experts with Nash Bargainingcited by 2
  8. 2026YuriiFormer: A Suite of Nesterov-Accelerated Transformerscited by 2
  9. 2024Demystifying the Token Dynamics of Deep Selective State Space Modelscited by 1
  10. 2026Second Order Drifting Modelscited by 1