# Aggregated Momentum: Stability Through Passive Damping

arXiv: https://arxiv.org/abs/1804.00325 · HTML: https://mlsanity.com/paper/1804.00325/

## Key facts

- Cited by 36 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 1 Apr 2018.

## Built on by (most cited first, 30 of 36)

- [Quasi-Global Momentum: Accelerating Decentralized Deep Learning on Heterogeneous Data](https://mlsanity.com/paper/2102.04761/) (2021)
- [The AdEMAMix Optimizer: Better, Faster, Older](https://mlsanity.com/paper/2409.03137/) (2024)
- [Practical tradeoffs between memory, compute, and performance in learned optimizers](https://mlsanity.com/paper/2203.11860/) (2022)
- [A Closer Look at Learned Optimization: Stability, Robustness, and Inductive Biases](https://mlsanity.com/paper/2209.11208/) (2022)
- [On the Limitations of Stochastic Pre-processing Defenses](https://mlsanity.com/paper/2206.09491/) (2022)
- [Improving physics-informed neural networks with meta-learned optimization](https://mlsanity.com/paper/2303.07127/) (2023)
- [Demystifying the Adversarial Robustness of Random Transformation Defenses](https://mlsanity.com/paper/2207.03574/) (2022)
- [Transformer-Based Learned Optimization](https://mlsanity.com/paper/2212.01055/) (2022)
- [Learning to Optimize for Reinforcement Learning](https://mlsanity.com/paper/2302.01470/) (2023)
- [Complex Momentum for Optimization in Games](https://mlsanity.com/paper/2102.08431/) (2021)
- [0/1 Deep Neural Networks via Block Coordinate Descent](https://mlsanity.com/paper/2206.09379/) (2022)
- [Exact Gauss-Newton Optimization for Training Deep Neural Networks](https://mlsanity.com/paper/2405.14402/) (2024)
- [Amortized Proximal Optimization](https://mlsanity.com/paper/2203.00089/) (2022)
- [Lookaround Optimizer: k steps around, 1 step average](https://mlsanity.com/paper/2306.07684/) (2023)
- [REX: Revisiting Budgeted Training with an Improved Schedule](https://mlsanity.com/paper/2107.04197/) (2021)
- [CoMMIT: Coordinated Multimodal Instruction Tuning](https://mlsanity.com/paper/2407.20454/) (2024)
- [Bidirectional Looking with A Novel Double Exponential Moving Average to Adaptive and Non-adaptive Momentum Optimizers](https://mlsanity.com/paper/2307.00631/) (2023)
- [Memory Augmented Optimizers for Deep Learning](https://mlsanity.com/paper/2106.10708/) (2021)
- [Diffusion Sampling with Momentum for Mitigating Divergence Artifacts](https://mlsanity.com/paper/2307.11118/) (2023)
- [AdaInject: Injection Based Adaptive Gradient Descent Optimizers for Convolutional Neural Networks](https://mlsanity.com/paper/2109.12504/) (2021)
- [BFE and AdaBFE: A New Approach in Learning Rate Automation for Stochastic Optimization](https://mlsanity.com/paper/2207.02763/) (2022)
- [Torque-Aware Momentum](https://mlsanity.com/paper/2412.18790/) (2024)
- [MT-DAO: Multi-Timescale Distributed Adaptive Optimizers with Local Updates](https://mlsanity.com/paper/2510.05361/) (2025)
- [RAL2M: Retrieval Augmented Learning-To-Match Against Hallucination in Compliance-Guaranteed Service Systems](https://mlsanity.com/paper/2601.02917/) (2026)
- [Compute Efficiency and Serial Runtime Tradeoffs for Stochastic Momentum Methods](https://mlsanity.com/paper/2606.19179/) (2026)
- [Distributed Task Management in Fog Computing: A Socially Concave Bandit Game](https://mlsanity.com/paper/2203.14572/) (2022)
- [Incremental Gauss-Newton Descent for Machine Learning](https://mlsanity.com/paper/2408.05560/) (2024)
- [Learning by solving differential equations](https://mlsanity.com/paper/2505.13397/) (2025)
- [A More Stable Accelerated Gradient Method Inspired by Continuous-Time Perspective](https://mlsanity.com/paper/2112.04922/) (2021)
- [Improved Binary Forward Exploration: Learning Rate Scheduling Method for Stochastic Optimization](https://mlsanity.com/paper/2207.04198/) (2022)

## Abstract (arXiv)

Momentum is a simple and widely used trick which allows gradient-based optimizers to pick up speed along low curvature directions. Its performance depends crucially on a damping coefficient β. Large β values can potentially deliver much larger speedups, but are prone to oscillations and instability; hence one typically resorts to small values such as 0.5 or 0.9. We propose Aggregated Momentum (AggMo), a variant of momentum which combines multiple velocity vectors with different β parameters. AggMo is trivial to implement, but significantly dampens oscillations, enabling it to remain stable even for aggressive β values such as 0.999. We reinterpret Nesterov's accelerated gradient descent as a special case of AggMo and analyze rates of convergence for quadratic objectives. Empirically, we find that AggMo is a suitable drop-in replacement for other momentum methods, and frequently delivers faster convergence.