mlsanityData to 14 Oct 2026

TiAda: A Time-scale Adaptive Algorithm for Nonconvex Minimax Optimization

arXiv 2210.17478 ↗First posted 31 Oct 2022math.OC cs.LG stat.MLcited by 8

Not on the map. Why?

Abstract (arXiv)

Adaptive gradient methods have shown their ability to adjust the stepsizes on the fly in a parameter-agnostic manner, and empirically achieve faster convergence for solving minimization problems. When it comes to nonconvex minimax optimization, however, current convergence analyses of gradient descent ascent (GDA) combined with adaptive stepsizes require careful tuning of hyper-parameters and the knowledge of problem-dependent parameters. Such a discrepancy arises from the primal-dual nature of minimax problems and the necessity of delicate time-scale separation between the primal and dual updates in attaining convergence. In this work, we propose a single-loop adaptive GDA algorithm called TiAda for nonconvex minimax optimization that automatically adapts to the time-scale separation. Our algorithm is fully parameter-agnostic and can achieve near-optimal complexities simultaneously in deterministic and stochastic settings of nonconvex-strongly-concave minimax problems. The effectiveness of the proposed method is further justified numerically for a number of machine learning applications.

Key facts. Cited by 8 papers in the map (data through 14 Oct 2026). Builds on 44 of its 69 references that are in the map. First posted on arXiv 31 Oct 2022.

Builds on

44 of 69 references.

References · 44

  1. 2014Adam: A Method for Stochastic Optimizationcited by 37,240
  2. 2014Explaining and Harnessing Adversarial Examplescited by 7,364
  3. 2015Domain-Adversarial Training of Neural Networkscited by 3,044
  4. 2016Improved Techniques for Training GANscited by 2,970
  5. 2017Improved Training of Wasserstein GANscited by 2,235
  6. 2020Sharpness-Aware Minimization for Efficiently Improving Generalizationcited by 1,417
  7. 2019On the Convergence of Adam and Beyondcited by 841
  8. 2017Sharp Minima Can Generalize For Deep Netscited by 466
  9. 2017Certifying Some Distributional Robustness with Principled Adversarial Trainingcited by 337
  10. 2016NIPS 2016 Tutorial: Generative Adversarial Networkscited by 285
  11. 2019On Gradient Descent Ascent for Nonconvex-Concave Minimax Problemscited by 270
  12. 2017Training GANs with Optimismcited by 217
32 more
  1. 2018AdaGrad stepsizes: Sharp convergence over nonconvex landscapescited by 182
  2. 2018On the Convergence of A Class of Adam-Type Algorithms for Non-Convex Optimizationcited by 174
  3. 2020A Simple Convergence Proof of Adam and Adagradcited by 171
  4. 2018On the Convergence of Stochastic Gradient Descent with Adaptive Stepsizescited by 156
  5. 2019Solving a Class of Non-Convex Min-Max Games Using Iterative First Order Methodscited by 148
  6. 2018A Variational Inequality Perspective on Generative Adversarial Networkscited by 144
  7. 2020Near-Optimal Algorithms for Minimax Optimizationcited by 119
  8. 2022Adam Can Converge Without Any Modification On Update Rulescited by 114
  9. 2020Robust Federated Learning: The Case of Affine Distribution Shiftscited by 102
  10. 2016Opponent Modeling in Deep Reinforcement Learningcited by 97
  11. 2018On the Convergence of Adaptive Gradient Methods for Nonconvex Optimizationcited by 91
  12. 2020The Complexity of Constrained Min-Max Optimizationcited by 82
  13. 2021Model-free Representation Learning and Exploration in Low-rank MDPscited by 82
  14. 2020A High Probability Analysis of Adaptive SGD with Momentumcited by 61
  15. 2019UniXGrad: A Universal, Adaptive Algorithm with Optimal Guarantees for Constrained Optimizationcited by 56
  16. 2018Online Adaptive Methods, Universality and Accelerationcited by 51
  17. 2021A Novel Convergence Analysis for Algorithms of the Adam Familycited by 50
  18. 2021Faster Single-loop Algorithms for Minimax Optimization without Strong Concavitycited by 49
  19. 2021The Complexity of Nonconvex-Strongly-Concave Minimax Optimizationcited by 45
  20. 2022High Probability Bounds for a Class of Nonconvex Algorithms with AdaGrad Stepsizecited by 40
  21. 2017Online to Offline Conversions, Universality and Adaptive Minibatch Sizescited by 39
  22. 2021Complexity Lower Bounds for Nonconvex-Strongly-Concave Min-Max Optimizationcited by 36
  23. 2019Revisiting Stochastic Extragradientcited by 35
  24. 2019Escaping Saddle Points with Adaptive Gradient Methodscited by 34
  25. 2019Linear Convergence of Adaptive Stochastic Gradient Descentcited by 29
  26. 2019A Universal Algorithm for Variational Inequalities Adaptive to Smoothness and Noisecited by 26
  27. 2020Alternating proximal-gradient steps for (stochastic) nonconvex-concave minimax problemscited by 26
  28. 2022SAPD+: An Accelerated Stochastic Method for Nonconvex-Concave Minimax Problemscited by 23
  29. 2022Nest Your Adaptive Algorithm for Parameter-Agnostic Nonconvex Minimax Optimizationcited by 16
  30. 2021AdaGDA: Faster Adaptive Gradient Descent Ascent Methods for Minimax Optimizationcited by 13
  31. 2022On Convergence of Gradient Descent Ascent: A Tight Local Analysiscited by 9
  32. 2021Randomized Stochastic Gradient Descent Ascentcited by 3

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 1; 2024: 3; 2025: 2; 2026: 2.

8 papers.

  1. 2024Two-Timescale Gradient Descent Ascent Algorithms for Nonconvex Minimax Optimizationcited by 16
  2. 2025An Optimisation Framework for Unsupervised Environment Designcited by 10
  3. 2023Enhanced Adaptive Gradient Algorithms for Nonconvex-PL Minimax Optimizationcited by 6
  4. 2025Adaptive Algorithms with Sharp Convergence Rates for Stochastic Hierarchical Optimizationcited by 2
  5. 2024Achieving Near-Optimal Convergence for Distributed Minimax Optimization with Adaptive Stepsizescited by 1
  6. 2024Completely Parameter-Free Single-Loop Algorithms for Nonconvex-Concave Minimax Problemscited by 0
  7. 2026Understanding Dynamics of Adam in Zero-Sum Games: An ODE Approachcited by 0
  8. 2026The Role of Gradient Modification in Heavy-Tailed Nonconvex Stochastic Min-Max Optimizationcited by 0