mlsanityData to 14 Oct 2026

Momentum Residual Neural Networks

arXiv 2102.07870 ↗First posted 15 Feb 2021cs.LG cs.AI stat.MLcited by 37

Not on the map. Why?

Abstract (arXiv)

The training of deep residual neural networks (ResNets) with backpropagation has a memory cost that increases linearly with respect to the depth of the network. A way to circumvent this issue is to use reversible architectures. In this paper, we propose to change the forward rule of a ResNet by adding a momentum term. The resulting networks, momentum residual neural networks (Momentum ResNets), are invertible. Unlike previous invertible architectures, they can be used as a drop-in replacement for any existing ResNet block. We show that Momentum ResNets can be interpreted in the infinitesimal step size regime as second-order ordinary differential equations (ODEs) and exactly characterize how adding momentum progressively increases the representation capabilities of Momentum ResNets. Our analysis reveals that Momentum ResNets can learn any linear mapping up to a multiplicative factor, while ResNets cannot. In a learning to optimize setting, where convergence to a fixed point is required, we show theoretically and empirically that our method succeeds while existing invertible architectures fail. We show on CIFAR and ImageNet that Momentum ResNets have the same accuracy as ResNets, while having a much smaller memory footprint, and show that pre-trained Momentum ResNets are promising for fine-tuning models.

Key facts. Cited by 37 papers in the map (data through 14 Oct 2026). Builds on 35 of its 63 references that are in the map. First posted on arXiv 15 Feb 2021.

Builds on

35 of 63 references.

References · 35

  1. 2015Deep Residual Learning for Image Recognitioncited by 52,096
  2. 2019Momentum Contrast for Unsupervised Visual Representation Learningcited by 7,513
  3. 2018Neural Ordinary Differential Equationscited by 3,779
  4. 2016An overview of gradient descent optimization algorithmscited by 1,033
  5. 2015Automatic differentiation in machine learning: a surveycited by 855
  6. 2019Big Transfer (BiT): General Visual Representation Learningcited by 769
  7. 2016TensorFlow: Large-Scale Machine Learning on Heterogeneous Distributed Systemscited by 664
  8. 2019Augmented Neural ODEscited by 397
  9. 2018Invertible Residual Networkscited by 365
  10. 2015Gradient-based Hyperparameter Optimization through Reversible Learningcited by 360
  11. 2017Stable Architectures for Deep Neural Networkscited by 341
  12. 2017Convolutional Neural Networks for Medical Image Analysis: Full Training or Fine Tuning?cited by 221
23 more
  1. 2017The Reversible Residual Network: Backpropagation Without Storing Activationscited by 217
  2. 2017Large Kernel Matters -- Improve Semantic Segmentation by Global Convolutional Networkcited by 208
  3. 2017Beyond Finite Layer Neural Networks: Bridging Deep Architectures and Numerical Differential Equationscited by 203
  4. 2018Deep Neural Networks Motivated by Partial Differential Equationscited by 187
  5. 2020Dissecting Neural ODEscited by 153
  6. 2018i-RevNet: Deep Invertible Networkscited by 137
  7. 2019ANODE: Unconditionally Accurate Memory-Efficient Gradients for Neural ODEscited by 84
  8. 2020Coupled Oscillatory Recurrent Neural Network (coRNN): An accurate and (gradient) stable architecture for learning long time dependenciescited by 82
  9. 2019Deep Learning via Dynamical Systems: An Approximation Perspectivecited by 78
  10. 2017Reversible Architectures for Arbitrarily Deep Residual Neural Networkscited by 70
  11. 2020On Second Order Behaviour in Augmented Neural ODEscited by 68
  12. 2018A Mean-Field Optimal Control Formulation of Deep Learningcited by 67
  13. 2019Approximation Capabilities of Neural ODEs and Invertible Residual Networkscited by 56
  14. 2018SuperNeurons: Dynamic GPU Memory Management for Training Deep Neural Networkscited by 55
  15. 2020Continuous-in-Depth Neural Networkscited by 40
  16. 2019Momentum-Net: Fast and convergent iterative neural network for inverse problemscited by 38
  17. 2020Universal Approximation Property of Neural Ordinary Differential Equationscited by 32
  18. 2020Fixing the train-test resolution discrepancy: FixEfficientNetcited by 31
  19. 2020MomentumRNN: Integrating Momentum into Recurrent Neural Networkscited by 18
  20. 2018Optimization Algorithm Inspired Deep Neural Network Structure Designcited by 14
  21. 2018Stochastic Training of Residual Networks: a Differential Equation Viewpointcited by 12
  22. 2019ANODEV2: A Coupled Neural ODE Evolution Frameworkcited by 12
  23. 2020Towards Understanding Normalization in Neural ODEscited by 7

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2021: 10; 2022: 8; 2023: 5; 2024: 5; 2025: 7; 2026: 2.

37 papers.

Sort:

By area:

  1. 2021Sinkformers: Transformers with Doubly Stochastic Attentioncited by 138
  2. 2021Efficient and Accurate Gradients for Neural SDEscited by 71
  3. 2021A Dynamical System Perspective for Lipschitz Neural Networkscited by 60
  4. 2021Heavy Ball Neural Ordinary Differential Equationscited by 42
  5. 2023Reversible Vision Transformerscited by 33
  6. 2021Improving Transformers with Probabilistic Attention Keyscited by 28
  7. 2022ODE Transformer: An Ordinary Differential Equation-Inspired Model for Sequence Generationcited by 27
  8. 2022Do Residual Neural Networks discretize Neural Ordinary Differential Equations?cited by 27
  9. 2021SHINE: SHaring the INverse Estimate from the forward pass for bi-level optimization and implicit modelscited by 26
  10. 2022Semi-supervised Invertible Neural Operators for Bayesian Inverse Problemscited by 25
  11. 2023TorchDEQ: A Library for Deep Equilibrium Modelscited by 21
  12. 2021How Does Momentum Benefit Deep Neural Networks Architecture Design? A Few Case Studiescited by 10
25 more
  1. 2022Neural ODE Control for Trajectory Approximation of Continuity Equationcited by 9
  2. 2024Efficient, Accurate and Stable Gradients for Neural ODEscited by 9
  3. 2024Understanding the training of infinitely deep and wide ResNets with Conditional Optimal Transportcited by 8
  4. 2021Can convolutional ResNets approximately preserve input distances? A frequency analysis perspectivecited by 7
  5. 2022RevBiFPN: The Fully Reversible Bidirectional Feature Pyramid Networkcited by 6
  6. 2023Memory-Efficient Reversible Spiking Neural Networkscited by 6
  7. 2021Neural Architecture Search via Bregman Iterationscited by 5
  8. 2022Robustness to corruption in pre-trained Bayesian neural networkscited by 5
  9. 2025Reversible Deep Equilibrium Modelscited by 5
  10. 2024Predictor-Corrector Enhanced Transformers with Exponential Moving Average Coefficient Learningcited by 4
  11. 2025Random Feature Representation Boostingcited by 4
  12. 2023Variational formulations of ODE-Net as a mean-field optimal control problem and existence resultscited by 3
  13. 2025Transformers as Intrinsic Optimizers: Forward Inference through the Energy Principlecited by 3
  14. 2022DASS: Differentiable Architecture Search for Sparse neural networkscited by 2
  15. 2023PaReprop: Fast Parallelized Reversible Backpropagationcited by 1
  16. 2024On Exact Bit-level Reversible Transformers Without Changing Architecturescited by 1
  17. 2025Control, Optimal Transport and Neural Differential Equations in Supervised Learningcited by 1
  18. 2026Efficient Unrolled Networks for Large-Scale 3D Inverse Problemscited by 1
  19. 2021HeunNet: Extending ResNet using Heun's Methodscited by 0
  20. 2022Momentum Capsule Networkscited by 0
  21. 2024A Novel Momentum-Based Deep Learning Techniques for Medical Image Classification and Segmentationcited by 0
  22. 2025Revisiting Reset Mechanisms in Spiking Neural Networks for Sequential Modeling: Specialized Discretization for Binary Activated RNNcited by 0
  23. 2025ParaRevSNN: A Parallel Reversible Spiking Neural Network for Efficient Training and Inferencecited by 0
  24. 2025IIET: Efficient Numerical Transformer via Implicit Iterative Euler Methodcited by 0
  25. 2026Deep learning from a control vantage pointcited by 0