# A Closer Look at the Optimization Landscapes of Generative Adversarial Networks

arXiv: https://arxiv.org/abs/1906.04848 · HTML: https://mlsanity.com/paper/1906.04848/

## Key facts

- Cited by 25 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 11 Jun 2019.

## Built on by (most cited first, 25 of 25)

- [CoLa-Diff: Conditional Latent Diffusion Model for Multi-Modal MRI Synthesis](https://mlsanity.com/paper/2303.14081/) (2023)
- [Fair Normalizing Flows](https://mlsanity.com/paper/2106.05937/) (2021)
- [ExtremeCast: Boosting Extreme Value Prediction for Global Weather Forecast](https://mlsanity.com/paper/2402.01295/) (2024)
- [On the Convergence of Stochastic Extragradient for Bilinear Games using Restarted Iteration Averaging](https://mlsanity.com/paper/2107.00464/) (2021)
- [Understanding Overparameterization in Generative Adversarial Networks](https://mlsanity.com/paper/2104.05605/) (2021)
- [SAN: Inducing Metrizability of GAN with Discriminative Normalized Linear Layer](https://mlsanity.com/paper/2301.12811/) (2023)
- [Complex Momentum for Optimization in Games](https://mlsanity.com/paper/2102.08431/) (2021)
- [Last-Iterate Convergence of Saddle-Point Optimizers via High-Resolution Differential Equations](https://mlsanity.com/paper/2112.13826/) (2021)
- [Phy-Diff: Physics-guided Hourglass Diffusion Model for Diffusion MRI Synthesis](https://mlsanity.com/paper/2406.03002/) (2024)
- [Fair Representations by Compression](https://mlsanity.com/paper/2105.14044/) (2021)
- [Domain Adversarial Training: A Game Perspective](https://mlsanity.com/paper/2202.05352/) (2022)
- [Minimax Optimality (Probably) Doesn't Imply Distribution Learning for GANs](https://mlsanity.com/paper/2201.07206/) (2022)
- [Multi-Resolution Continuous Normalizing Flows](https://mlsanity.com/paper/2106.08462/) (2021)
- [Fixed-kinetic Neural Hamiltonian Flows for enhanced interpretability and reduced complexity](https://mlsanity.com/paper/2302.01955/) (2023)
- [A Differential Geometry Perspective on Orthogonal Recurrent Models](https://mlsanity.com/paper/2102.09589/) (2021)
- [Generative Minimization Networks: Training GANs Without Competition](https://mlsanity.com/paper/2103.12685/) (2021)
- [On Characterizing GAN Convergence Through Proximal Duality Gap](https://mlsanity.com/paper/2105.04801/) (2021)
- [Relaxed Conditional Image Transfer for Semi-supervised Domain Adaptation](https://mlsanity.com/paper/2101.01400/) (2021)
- [Generative Adversarial Network: Some Analytical Perspectives](https://mlsanity.com/paper/2104.12210/) (2021)
- [Convergence of GANs Training: A Game and Stochastic Control Methodology](https://mlsanity.com/paper/2112.00222/) (2021)
- [Generative Adversarial Networks via a Composite Annealing of Noise and Diffusion](https://mlsanity.com/paper/2105.00220/) (2021)
- [When is Momentum Extragradient Optimal? A Polynomial-Based Analysis](https://mlsanity.com/paper/2211.04659/) (2022)
- [Local convergence of simultaneous min-max algorithms to differential equilibrium on Riemannian manifold](https://mlsanity.com/paper/2405.13392/) (2024)
- [Towards Scalable Topological Regularizers](https://mlsanity.com/paper/2501.14641/) (2025)
- [On the Nash equilibrium of moment-matching GANs for stationary Gaussian processes](https://mlsanity.com/paper/2203.07136/) (2022)

## Abstract (arXiv)

Generative adversarial networks have been very successful in generative modeling, however they remain relatively challenging to train compared to standard deep neural networks. In this paper, we propose new visualization techniques for the optimization landscapes of GANs that enable us to study the game vector field resulting from the concatenation of the gradient of both players. Using these visualization techniques we try to bridge the gap between theory and practice by showing empirically that the training of GANs exhibits significant rotations around Local Stable Stationary Points (LSSP), similar to the one predicted by theory on toy examples. Moreover, we provide empirical evidence that GAN training converge to a stable stationary point which is a saddle point for the generator loss, not a minimum, while still achieving excellent performance.