mlsanityData to 7 Oct 2026

Gradient descent GAN optimization is locally stable

arXiv 1706.04156 ↗First posted 13 Jun 2017cs.LG cs.AI math.OCcited by 65

Not on the map. Why?

Abstract (arXiv)

Despite the growing prominence of generative adversarial networks (GANs), optimization in GANs is still a poorly understood topic. In this paper, we analyze the "gradient descent" form of GAN optimization i.e., the natural setting where we simultaneously take small gradient steps in both generator and discriminator parameters. We show that even though GAN optimization does not correspond to a convex-concave game (even for simple parameterizations), under proper conditions, equilibrium points of this optimization procedure are still locally asymptotically stable for the traditional GAN formulation. On the other hand, we show that the recently proposed Wasserstein GAN can have non-convergent limit cycles near equilibrium. Motivated by this stability analysis, we propose an additional regularization term for gradient descent GAN updates, which is able to guarantee local stability for both the WGAN and the traditional GAN, and also shows practical promise in speeding up convergence and addressing mode collapse.

Key facts. Cited by 65 papers in the map (data through 7 Oct 2026). First posted on arXiv 13 Jun 2017.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 21; 2022: 14; 2023: 10; 2024: 7; 2025: 8; 2026: 5.

65 papers.

Sort:

By area:

  1. 2021Faster Single-loop Algorithms for Minimax Optimization without Strong Concavitycited by 47
  2. 2022Why neural networks find simple solutions: the many regularizers of geometric complexitycited by 38
  3. 2024Generative Artificial Intelligence: A Systematic Review and Applicationscited by 37
  4. 2023Free Lunch for Domain Adversarial Training: Environment Label Smoothingcited by 33
  5. 2024PaGoDA: Progressive Growing of a One-Step Generator from a Low-Resolution Diffusion Teachercited by 27
  6. 2021A Neural Tangent Kernel Perspective of GANscited by 21
  7. 2022Improving 3D-aware Image Synthesis with A Geometry-aware Discriminatorcited by 16
  8. 2021Understanding Overparameterization in Generative Adversarial Networkscited by 14
  9. 2021Discretization Drift in Two-Player Gamescited by 14
  10. 2023SAN: Inducing Metrizability of GAN with Discriminative Normalized Linear Layercited by 14
  11. 2022Max-Min Off-Policy Actor-Critic Method Focusing on Worst-Case Robustness to Model Misspecificationcited by 11
  12. 2023On a continuous time model of gradient descent dynamics and instability in deep learningcited by 11
38 more
  1. 2022Dual-Scale Single Image Dehazing Via Neural Augmentationcited by 10
  2. 2021Follow-the-Regularized-Leader Routes to Chaos in Routing Gamescited by 9
  3. 2023Synthetic Health-related Longitudinal Data with Mixed-type Variables Generated using Diffusion Modelscited by 9
  4. 2021Minimax Optimization with Smooth Algorithmic Adversariescited by 8
  5. 2021Generalization Error of GAN from the Discriminator's Perspectivecited by 8
  6. 2022Existence and Estimation of Critical Batch Size for Training Generative Adversarial Networks with Two Time-Scale Update Rulecited by 8
  7. 2023Nash Equilibria, Regularization and Computation in Optimal Transport-Based Distributionally Robust Optimizationcited by 8
  8. 2023Analyzing and Improving Optimal-Transport-based Adversarial Networkscited by 8
  9. 2021Adversarial Deep Learning for Online Resource Allocationcited by 7
  10. 2021The Geometric Occam's Razor Implicit in Deep Learningcited by 7
  11. 2022Instability and Local Minima in GAN Training with Kernel Discriminatorscited by 7
  12. 2022A Mathematical Framework for Learning Probability Distributionscited by 7
  13. 2022An Efficient Stochastic Algorithm for Decentralized Nonconvex-Strongly-Concave Minimax Optimizationcited by 6
  14. 2021Multi-View Feature Representation for Dialogue Generation with Bidirectional Distillationcited by 5
  15. 2021A Cubic Regularization Approach for Finding Local Minimax Points in Nonconvex Minimax Optimizationcited by 5
  16. 2021Functional Space Analysis of Local GAN Convergencecited by 4
  17. 2021Synthetic flow-based cryptomining attack generation through Generative Adversarial Networkscited by 4
  18. 2021Generative Adversarial Networks and Adversarial Autoencoders: Tutorial and Surveycited by 4
  19. 2022Optimal Extragradient-Based Bilinearly-Coupled Saddle-Point Optimizationcited by 4
  20. 2021Statistical inference for generative adversarial networks and other minimax problemscited by 3
  21. 2021Generalization of GANs and overparameterized models under Lipschitz continuitycited by 2
  22. 2021Adversarial Representation Learning With Closed-Form Solverscited by 2
  23. 2021Generative Adversarial Network for Probabilistic Forecast of Random Dynamical Systemcited by 2
  24. 2022Single-level Adversarial Data Synthesis based on Neural Tangent Kernelscited by 2
  25. 2022Generative Data Augmentation for Non-IID Problem in Decentralized Clinical Machine Learningcited by 2
  26. 2023Mind the (optimality) Gap: A Gap-Aware Learning Rate Scheduler for Adversarial Netscited by 2
  27. 2025A Unified Framework for Diffusion Model Unlearning with f-Divergencecited by 2
  28. 2021Generative Adversarial Networks via a Composite Annealing of Noise and Diffusioncited by 1
  29. 2021Stochastic Projective Splitting: Solving Saddle-Point Problems with Multiple Regularizerscited by 1
  30. 2021Improving the quality of generative models through Smirnov transformationcited by 1
  31. 2022Continuous-time Analysis for Variational Inequalities: An Overview and Desideratacited by 1
  32. 2024On the Nonconvexity of Push-Forward Constraints and Its Consequences in Machine Learningcited by 1
  33. 2024Control of Overfitting with Physicscited by 1
  34. 2025General Autonomous Cybersecurity Defense: Learning Robust Policies for Dynamic Topologies and Diverse Attackerscited by 1
  35. 2025StegOT: Trade-offs in Steganography via Optimal Transportcited by 1
  36. 2026RelA-Diffusion: Relativistic Adversarial Diffusion for Multi-Tracer PET Synthesis from Multi-Sequence MRIcited by 1
  37. 2022On the Nash equilibrium of moment-matching GANs for stationary Gaussian processescited by 0
  38. 2022Dynamics of Fourier Modes in Torus Generative Adversarial Networkscited by 0

50 most cited of 65 All 65 (JSON)