mlsanityData to 14 Oct 2026

Aggregated Momentum: Stability Through Passive Damping

arXiv 1804.00325 ↗First posted 1 Apr 2018cs.LG cs.AI math.OCcited by 36

Not on the map. Why?

Abstract (arXiv)

Momentum is a simple and widely used trick which allows gradient-based optimizers to pick up speed along low curvature directions. Its performance depends crucially on a damping coefficient β. Large β values can potentially deliver much larger speedups, but are prone to oscillations and instability; hence one typically resorts to small values such as 0.5 or 0.9. We propose Aggregated Momentum (AggMo), a variant of momentum which combines multiple velocity vectors with different β parameters. AggMo is trivial to implement, but significantly dampens oscillations, enabling it to remain stable even for aggressive β values such as 0.999. We reinterpret Nesterov's accelerated gradient descent as a special case of AggMo and analyze rates of convergence for quadratic objectives. Empirically, we find that AggMo is a suitable drop-in replacement for other momentum methods, and frequently delivers faster convergence.

Key facts. Cited by 36 papers in the map (data through 14 Oct 2026). First posted on arXiv 1 Apr 2018.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2021: 6; 2022: 11; 2023: 6; 2024: 6; 2025: 2; 2026: 5.

36 papers.

Sort:

By area:

  1. 2021Quasi-Global Momentum: Accelerating Decentralized Deep Learning on Heterogeneous Datacited by 69
  2. 2024The AdEMAMix Optimizer: Better, Faster, Oldercited by 54
  3. 2022Practical tradeoffs between memory, compute, and performance in learned optimizerscited by 32
  4. 2022A Closer Look at Learned Optimization: Stability, Robustness, and Inductive Biasescited by 30
  5. 2022On the Limitations of Stochastic Pre-processing Defensescited by 21
  6. 2023Improving physics-informed neural networks with meta-learned optimizationcited by 19
  7. 2022Demystifying the Adversarial Robustness of Random Transformation Defensescited by 14
  8. 2022Transformer-Based Learned Optimizationcited by 13
  9. 2023Learning to Optimize for Reinforcement Learningcited by 12
  10. 2021Complex Momentum for Optimization in Gamescited by 11
  11. 20220/1 Deep Neural Networks via Block Coordinate Descentcited by 10
  12. 2024Exact Gauss-Newton Optimization for Training Deep Neural Networkscited by 10
24 more
  1. 2022Amortized Proximal Optimizationcited by 9
  2. 2023Lookaround Optimizer: k steps around, 1 step averagecited by 9
  3. 2021REX: Revisiting Budgeted Training with an Improved Schedulecited by 8
  4. 2024CoMMIT: Coordinated Multimodal Instruction Tuningcited by 8
  5. 2023Bidirectional Looking with A Novel Double Exponential Moving Average to Adaptive and Non-adaptive Momentum Optimizerscited by 6
  6. 2021Memory Augmented Optimizers for Deep Learningcited by 5
  7. 2023Diffusion Sampling with Momentum for Mitigating Divergence Artifactscited by 3
  8. 2021AdaInject: Injection Based Adaptive Gradient Descent Optimizers for Convolutional Neural Networkscited by 2
  9. 2022BFE and AdaBFE: A New Approach in Learning Rate Automation for Stochastic Optimizationcited by 2
  10. 2024Torque-Aware Momentumcited by 2
  11. 2025MT-DAO: Multi-Timescale Distributed Adaptive Optimizers with Local Updatescited by 2
  12. 2026RAL2M: Retrieval Augmented Learning-To-Match Against Hallucination in Compliance-Guaranteed Service Systemscited by 2
  13. 2026Compute Efficiency and Serial Runtime Tradeoffs for Stochastic Momentum Methodscited by 2
  14. 2022Distributed Task Management in Fog Computing: A Socially Concave Bandit Gamecited by 1
  15. 2024Incremental Gauss-Newton Descent for Machine Learningcited by 1
  16. 2025Learning by solving differential equationscited by 1
  17. 2021A More Stable Accelerated Gradient Method Inspired by Continuous-Time Perspectivecited by 0
  18. 2022Improved Binary Forward Exploration: Learning Rate Scheduling Method for Stochastic Optimizationcited by 0
  19. 2022Multilevel-in-Layer Training for Deep Neural Network Regressioncited by 0
  20. 2023Fine-Tuning Adaptive Stochastic Optimizers: Determining the Optimal Hyperparameter ε via Gradient Magnitude Histogram Analysiscited by 0
  21. 2024Neural Optimizer Equation, Decay Function, and Learning Rate Schedule Joint Evolutioncited by 0
  22. 2026OmniOpt: Taxonomy, Geometry, and Benchmarking of Modern Optimizerscited by 0
  23. 2026Curvature-Conditioned Multiscale Momentum with Sphere Constraints for LLM Pretrainingcited by 0
  24. 2026A Journey to the Edge of Stabilitycited by 0