mlsanityData to 14 Oct 2026

Continuous-Time Analysis of Heavy Ball Momentum in Min-Max Games

arXiv 2505.19537 ↗First posted 26 May 2025cs.GT cs.LGcited by 3

Not on the map. Why?

Abstract (arXiv)

Since Polyak's pioneering work, heavy ball (HB) momentum has been widely studied in minimization. However, its role in min-max games remains largely unexplored. As a key component of practical min-max algorithms like Adam, this gap limits their effectiveness. In this paper, we present a continuous-time analysis for HB with simultaneous and alternating update schemes in min-max games. Locally, we prove smaller momentum enhances algorithmic stability by enabling local convergence across a wider range of step sizes, with alternating updates generally converging faster. Globally, we study the implicit regularization of HB, and find smaller momentum guides algorithms trajectories towards shallower slope regions of the loss landscapes, with alternating updates amplifying this effect. Surprisingly, all these phenomena differ from those observed in minimization, where larger momentum yields similar effects. Our results reveal fundamental differences between HB in min-max games and minimization, and numerical experiments further validate our theoretical results.

Key facts. Cited by 3 papers in the map (data through 14 Oct 2026). Builds on 39 of its 56 references that are in the map. First posted on arXiv 26 May 2025.

Builds on

39 of 56 references.

References · 39

  1. 2017Towards Deep Learning Models Resistant to Adversarial Attackscited by 6,415
  2. 2016Improved Techniques for Training GANscited by 2,970
  3. 2017Improved Training of Wasserstein GANscited by 2,235
  4. 2020Sharpness-Aware Minimization for Efficiently Improving Generalizationcited by 1,417
  5. 2018Which Training Methods for GANs do actually Converge?cited by 679
  6. 2017Training GANs with Optimismcited by 217
  7. 2015A Differential Equation for Modeling Nesterov's Accelerated Gradient Method: Theory and Insightscited by 206
  8. 2020Implicit Gradient Regularizationcited by 146
  9. 2015Stochastic modified equations and adaptive stochastic gradient algorithmscited by 142
  10. 2021On the Convergence and Robustness of Adversarial Trainingcited by 130
  11. 2016A Variational Perspective on Accelerated Methods in Optimizationcited by 110
  12. 2018Understanding the Acceleration Phenomenon via High-Resolution Differential Equationscited by 69
27 more
  1. 2018Negative Momentum for Improved Game Dynamicscited by 60
  2. 2020The limits of min-max optimization algorithms: convergence to spurious non-critical setscited by 50
  3. 2021Faster Single-loop Algorithms for Minimax Optimization without Strong Concavitycited by 49
  4. 2019Differentiable Game Mechanicscited by 43
  5. 2019A Dynamical Systems Perspective on Nesterov Accelerationcited by 33
  6. 2019Continuous Time Analysis of Momentum Methodscited by 32
  7. 2021Near-optimal Local Convergence of Alternating Gradient Descent-Ascent for Minimax Optimizationcited by 31
  8. 2022Momentum Adversarial Distillation: Handling Large Distribution Shifts in Data-Free Knowledge Distillationcited by 31
  9. 2019Finite Regret and Cycles with Fixed Step-Size via Alternating Gradient Descent-Ascentcited by 29
  10. 2020Optimization with Momentum: Dynamical, Control-Theoretic, and Symplectic Perspectivescited by 27
  11. 2021Gradient Normalization for Generative Adversarial Networkscited by 25
  12. 2023Implicit regularization in Heavy-ball momentum accelerated stochastic gradient descentcited by 25
  13. 2024Adaptive Methods through the Lens of SDEs: Theoretical Insights on the Role of Noisecited by 23
  14. 2014On the Characterization of Local Nash Equilibria in Continuous Gamescited by 22
  15. 2020Accelerating Smooth Games by Manipulating Spectral Shapescited by 18
  16. 2021Discretization Drift in Two-Player Gamescited by 14
  17. 2022Alternating Mirror Descent for Constrained Min-Max Gamescited by 13
  18. 2020On the Suboptimality of Negative Momentum for Minimax Optimizationcited by 11
  19. 2022Provable Acceleration of Heavy Ball beyond Quadratics for a Class of Polyak-Lojasiewicz Functions when the Non-Convexity is Averaged-Outcited by 11
  20. 2016Local Convergence of the Heavy-ball Method and iPiano for Non-convex Optimizationcited by 9
  21. 2020A Unified Analysis of First-Order Methods for Smooth Games via Integral Quadratic Constraintscited by 9
  22. 2024Fundamental Benefit of Alternating Updates in Minimax Optimizationcited by 9
  23. 2024SDEs for Minimax Optimizationcited by 6
  24. 2025Alternating Regret for Online Convex Optimizationcited by 4
  25. 2023Stable Nonconvex-Nonconcave Training via Linear Interpolationcited by 3
  26. 2024Prediction Accuracy of Learning in Games : Follow-the-Regularized-Leader meets Heisenbergcited by 2
  27. 2024Accelerated regularized learning in finite N-person gamescited by 2

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 3.

3 papers.

  1. 2026Optimal Alternating Regret for Online Learning and Gamescited by 2
  2. 2026Negative Momentum for Convex-Concave Optimizationcited by 0
  3. 2026Understanding Dynamics of Adam in Zero-Sum Games: An ODE Approachcited by 0