# Gradient descent GAN optimization is locally stable

arXiv: https://arxiv.org/abs/1706.04156 · HTML: https://mlsanity.com/paper/1706.04156/

## Key facts

- Cited by 67 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 13 Jun 2017.

## Built on by (most cited first, 30 of 67)

- [Faster Single-loop Algorithms for Minimax Optimization without Strong Concavity](https://mlsanity.com/paper/2112.05604/) (2021)
- [DiffuseLoco: Real-Time Legged Locomotion Control with Diffusion from Offline Datasets](https://mlsanity.com/paper/2404.19264/) (2024)
- [Why neural networks find simple solutions: the many regularizers of geometric complexity](https://mlsanity.com/paper/2209.13083/) (2022)
- [Generative Artificial Intelligence: A Systematic Review and Applications](https://mlsanity.com/paper/2405.11029/) (2024)
- [Free Lunch for Domain Adversarial Training: Environment Label Smoothing](https://mlsanity.com/paper/2302.00194/) (2023)
- [PaGoDA: Progressive Growing of a One-Step Generator from a Low-Resolution Diffusion Teacher](https://mlsanity.com/paper/2405.14822/) (2024)
- [A Neural Tangent Kernel Perspective of GANs](https://mlsanity.com/paper/2106.05566/) (2021)
- [Improving 3D-aware Image Synthesis with A Geometry-aware Discriminator](https://mlsanity.com/paper/2209.15637/) (2022)
- [Understanding Overparameterization in Generative Adversarial Networks](https://mlsanity.com/paper/2104.05605/) (2021)
- [Discretization Drift in Two-Player Games](https://mlsanity.com/paper/2105.13922/) (2021)
- [SAN: Inducing Metrizability of GAN with Discriminative Normalized Linear Layer](https://mlsanity.com/paper/2301.12811/) (2023)
- [Max-Min Off-Policy Actor-Critic Method Focusing on Worst-Case Robustness to Model Misspecification](https://mlsanity.com/paper/2211.03413/) (2022)
- [On a continuous time model of gradient descent dynamics and instability in deep learning](https://mlsanity.com/paper/2302.01952/) (2023)
- [Synthetic Health-related Longitudinal Data with Mixed-type Variables Generated using Diffusion Models](https://mlsanity.com/paper/2303.12281/) (2023)
- [Follow-the-Regularized-Leader Routes to Chaos in Routing Games](https://mlsanity.com/paper/2102.07974/) (2021)
- [Dual-Scale Single Image Dehazing Via Neural Augmentation](https://mlsanity.com/paper/2209.05913/) (2022)
- [Minimax Optimization with Smooth Algorithmic Adversaries](https://mlsanity.com/paper/2106.01488/) (2021)
- [Generalization Error of GAN from the Discriminator's Perspective](https://mlsanity.com/paper/2107.03633/) (2021)
- [Existence and Estimation of Critical Batch Size for Training Generative Adversarial Networks with Two Time-Scale Update Rule](https://mlsanity.com/paper/2201.11989/) (2022)
- [Nash Equilibria, Regularization and Computation in Optimal Transport-Based Distributionally Robust Optimization](https://mlsanity.com/paper/2303.03900/) (2023)
- [Analyzing and Improving Optimal-Transport-based Adversarial Networks](https://mlsanity.com/paper/2310.02611/) (2023)
- [Adversarial Deep Learning for Online Resource Allocation](https://mlsanity.com/paper/2111.10285/) (2021)
- [The Geometric Occam's Razor Implicit in Deep Learning](https://mlsanity.com/paper/2111.15090/) (2021)
- [Instability and Local Minima in GAN Training with Kernel Discriminators](https://mlsanity.com/paper/2208.09938/) (2022)
- [A Mathematical Framework for Learning Probability Distributions](https://mlsanity.com/paper/2212.11481/) (2022)
- [An Efficient Stochastic Algorithm for Decentralized Nonconvex-Strongly-Concave Minimax Optimization](https://mlsanity.com/paper/2212.02387/) (2022)
- [Multi-View Feature Representation for Dialogue Generation with Bidirectional Distillation](https://mlsanity.com/paper/2102.10780/) (2021)
- [A Cubic Regularization Approach for Finding Local Minimax Points in Nonconvex Minimax Optimization](https://mlsanity.com/paper/2110.07098/) (2021)
- [Functional Space Analysis of Local GAN Convergence](https://mlsanity.com/paper/2102.04448/) (2021)
- [Synthetic flow-based cryptomining attack generation through Generative Adversarial Networks](https://mlsanity.com/paper/2107.14776/) (2021)

## Abstract (arXiv)

Despite the growing prominence of generative adversarial networks (GANs), optimization in GANs is still a poorly understood topic. In this paper, we analyze the "gradient descent" form of GAN optimization i.e., the natural setting where we simultaneously take small gradient steps in both generator and discriminator parameters. We show that even though GAN optimization does not correspond to a convex-concave game (even for simple parameterizations), under proper conditions, equilibrium points of this optimization procedure are still locally asymptotically stable for the traditional GAN formulation. On the other hand, we show that the recently proposed Wasserstein GAN can have non-convergent limit cycles near equilibrium. Motivated by this stability analysis, we propose an additional regularization term for gradient descent GAN updates, which is able to guarantee local stability for both the WGAN and the traditional GAN, and also shows practical promise in speeding up convergence and addressing mode collapse.