mlsanityData to 7 Oct 2026

On Scaled Methods for Saddle Point Problems

arXiv 2206.08303 ↗First posted 16 Jun 2022cs.LG math.OCcited by 3

Not on the map. Why?

Abstract (arXiv)

Methods with adaptive scaling of different features play a key role in solving saddle point problems, primarily due to Adam's popularity for solving adversarial machine learning problems, including GANS training. This paper carries out a theoretical analysis of the following scaling techniques for solving SPPs: the well-known Adam and RmsProp scaling and the newer AdaHessian and OASIS based on Hutchison approximation. We use the Extra Gradient and its improved version with negative momentum as the basic method. Experimental studies on GANs show good applicability not only for Adam, but also for other less popular methods.

Key facts. Cited by 3 papers in the map (data through 7 Oct 2026). Builds on 25 of its 52 references that are in the map. First posted on arXiv 16 Jun 2022.

Builds on

25 of 52 references.

References · 25

  1. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  2. 2022Generative Adversarial Networkscited by 5,603
  3. 2016Improved Techniques for Training GANscited by 2,938
  4. 2019ALBERT: A Lite BERT for Self-supervised Learning of Language Representationscited by 2,772
  5. 2018Spectral Normalization for Generative Adversarial Networkscited by 1,618
  6. 2019On the Convergence of Adam and Beyondcited by 834
  7. 2016NIPS 2016 Tutorial: Generative Adversarial Networkscited by 284
  8. 2017Training GANs with Optimismcited by 217
  9. 2020ADAHESSIAN: An Adaptive Second Order Optimizer for Machine Learningcited by 201
  10. 2020A Simple Convergence Proof of Adam and Adagradcited by 169
  11. 2018A Variational Inequality Perspective on Generative Adversarial Networkscited by 144
  12. 2018Optimistic mirror descent in saddle-point problems: Going the extra (gradient) milecited by 132
13 more
  1. 2020Efficient Methods for Structured Nonconvex-Nonconcave Min-Max Optimizationcited by 94
  2. 2019On the convergence of single-call stochastic extra-gradient methodscited by 88
  3. 2018Lower complexity bounds of first-order methods for convex-concave bilinear saddle-point problemscited by 75
  4. 2018Interaction Matters: A Note on Non-asymptotic Local Convergence of Generative Adversarial Networkscited by 68
  5. 2019Reducing Noise in GAN Training with Variance Reduced Extragradientcited by 61
  6. 2018Negative Momentum for Improved Game Dynamicscited by 59
  7. 2020Explore Aggressively, Update Conservatively: Stochastic Extragradient Methods with Variable Stepsize Scalingcited by 49
  8. 2021Stochastic Variance Reduction for Variational Inequality Methodscited by 46
  9. 2019Towards Better Understanding of Adaptive Gradient Algorithms in Generative Adversarial Netscited by 29
  10. 2021Doubly Adaptive Scaled Algorithm for Machine Learning Using Second-Order Informationcited by 16
  11. 2022Optimal Algorithms for Decentralized Stochastic Variational Inequalitiescited by 14
  12. 2019Training GANs with Centripetal Accelerationcited by 13
  13. 2021On the One-sided Convergence of Adam-type Algorithms in Non-convex Non-concave Min-max Optimizationcited by 9

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 1; 2023: 0; 2024: 1; 2025: 0; 2026: 1.

3 papers.

  1. 2022Smooth Monotone Stochastic Variational Inequalities and Saddle Point Problems: A Surveycited by 10
  2. 2024Local Methods with Adaptivity via Scalingcited by 1
  3. 2026Negative Momentum for Convex-Concave Optimizationcited by 0