mlsanityData to 14 Oct 2026

Complex Momentum for Optimization in Games

arXiv 2102.08431 ↗First posted 16 Feb 2021cs.LG cs.GTcited by 11

Not on the map. Why?

Abstract (arXiv)

We generalize gradient descent with momentum for optimization in differentiable games to have complex-valued momentum. We give theoretical motivation for our method by proving convergence on bilinear zero-sum games for simultaneous and alternating updates. Our method gives real-valued parameter updates, making it a drop-in replacement for standard optimizers. We empirically demonstrate that complex-valued momentum can improve convergence in realistic adversarial games - like generative adversarial networks - by showing we can find better solutions with an almost identical computational cost. We also show a practical generalization to a complex-valued Adam variant, which we use to train BigGAN to better inception scores on CIFAR-10.

Key facts. Cited by 11 papers in the map (data through 14 Oct 2026). Builds on 59 of its 104 references that are in the map. First posted on arXiv 16 Feb 2021.

Builds on

59 of 104 references.

References · 59

  1. 2014Adam: A Method for Stochastic Optimizationcited by 37,240
  2. 2016Improved Techniques for Training GANscited by 2,970
  3. 2018Large Scale GAN Training for High Fidelity Natural Image Synthesiscited by 2,626
  4. 2018Which Training Methods for GANs do actually Converge?cited by 679
  5. 2018Meta-Learning for Semi-Supervised Few-Shot Classificationcited by 456
  6. 2015Gradient-based Hyperparameter Optimization through Reversible Learningcited by 360
  7. 2017Adversarial Examples: Attacks and Defenses for Deep Learningcited by 351
  8. 2019Lookahead Optimizer: k steps forward, 1 step backcited by 336
  9. 2019Emergent Tool Use From Multi-Agent Autocurriculacited by 313
  10. 2019Optimizing Millions of Hyperparameters by Implicit Differentiationcited by 304
  11. 2017Learning with Opponent-Learning Awarenesscited by 222
  12. 2017Training GANs with Optimismcited by 217
38 more
  1. 2016Unrolled Generative Adversarial Networkscited by 208
  2. 2013Optimization, Learning, and Games with Predictable Sequencescited by 188
  3. 2019Solving a Class of Non-Convex Min-Max Games Using Iterative First Order Methodscited by 148
  4. 2018A Variational Inequality Perspective on Generative Adversarial Networkscited by 144
  5. 2017Intrinsic Motivation and Automatic Curricula via Asymmetric Self-Playcited by 141
  6. 2017The Numerics of GANscited by 105
  7. 2019Self-Tuning Networks: Bilevel Optimization of Hyperparameters using Structured Best-Response Functionscited by 105
  8. 2019On Empirical Comparisons of Optimizers for Deep Learningcited by 103
  9. 2019Open-ended Learning in Symmetric Zero-sum Gamescited by 96
  10. 2018Stochastic Hyperparameter Optimization through Hypernetworkscited by 73
  11. 2020A Game Theoretic Framework for Model Based Reinforcement Learningcited by 73
  12. 2020Real World Games Look Like Spinning Topscited by 66
  13. 2019Reducing Noise in GAN Training with Variance Reduced Extragradientcited by 62
  14. 2020From Poincar'e Recurrence to Convergence in Imperfect Information Games: Finding Equilibrium via Regularizationcited by 61
  15. 2018Negative Momentum for Improved Game Dynamicscited by 60
  16. 2016Connecting Generative Adversarial Networks and Actor-Critic Methodscited by 50
  17. 2018Finding Mixed Nash Equilibria of Generative Adversarial Networkscited by 49
  18. 2019Differentiable Game Mechanicscited by 43
  19. 2019LOGAN: Latent Optimisation for Generative Adversarial Networkscited by 42
  20. 2018Large-Scale Distributed Second-Order Optimization Using Kronecker-Factored Approximate Curvature for Deep Convolutional Neural Networkscited by 37
  21. 2018Aggregated Momentum: Stability Through Passive Dampingcited by 36
  22. 2019On Solving Minimax Optimization Locally: A Follow-the-Ridge Approachcited by 34
  23. 2020Adversarial Example Gamescited by 34
  24. 2019Competitive Gradient Descentcited by 32
  25. 2020Stochastic Hamiltonian Gradient Methods for Smooth Gamescited by 32
  26. 2019Finite Regret and Cycles with Fixed Step-Size via Alternating Gradient Descent-Ascentcited by 29
  27. 2019Towards Better Understanding of Adaptive Gradient Algorithms in Generative Adversarial Netscited by 29
  28. 2019A Closer Look at the Optimization Landscapes of Generative Adversarial Networkscited by 25
  29. 2019Multi-objective training of Generative Adversarial Networks with multiple discriminatorscited by 24
  30. 2020Policy-Aware Model Learning for Policy Gradient Methodscited by 23
  31. 2020Training Generative Adversarial Networks by Solving Ordinary Differential Equationscited by 21
  32. 2020Accelerating Smooth Games by Manipulating Spectral Shapescited by 18
  33. 2017YellowFin and the Art of Momentum Tuningcited by 16
  34. 2020Teaching with Commentariescited by 16
  35. 2019Implicit competitive regularization in GANscited by 14
  36. 2019Training GANs with Centripetal Accelerationcited by 13
  37. 2019Understanding Neural Architecture Search Techniquescited by 12
  38. 2019Convergence of Gradient Methods on Bilinear Zero-Sum Gamescited by 12

50 most cited of 59 All 59 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2022: 4; 2023: 1; 2024: 2; 2025: 2; 2026: 2.

11 papers.

  1. 2023ATT3D: Amortized Text-to-3D Object Synthesiscited by 87
  2. 2022The Optimal Noise in Noise-Contrastive Learning Is Not What You Thinkcited by 11
  3. 2024MomentumSMoE: Integrating Momentum into Sparse Mixture of Expertscited by 9
  4. 2022Domain Adversarial Training: A Game Perspectivecited by 7
  5. 2022Task Selection for AutoML System Evaluationcited by 7
  6. 2024JacNet: Learning Functions with Structured Jacobianscited by 7
  7. 2025Score Distillation Sampling for Audio: Source Separation, Synthesis, and Beyondcited by 6
  8. 2026Motion Attribution for Video Generationcited by 5
  9. 2025Expert Merging in Sparse Mixture of Experts with Nash Bargainingcited by 3
  10. 2022When is Momentum Extragradient Optimal? A Polynomial-Based Analysiscited by 1
  11. 2026Negative Momentum for Convex-Concave Optimizationcited by 0