mlsanityData to 14 Oct 2026

Generalization of GANs and overparameterized models under Lipschitz continuity

arXiv 2104.02388 ↗First posted 6 Apr 2021cs.LGcited by 2

Not on the map. Why?

Abstract (arXiv)

Generative adversarial networks (GANs) are so complex that the existing learning theories do not provide a satisfactory explanation for why GANs have great success in practice. The same situation also remains largely open for deep neural networks. To fill this gap, we introduce a Lipschitz theory to analyze generalization. We demonstrate its simplicity by analyzing generalization and consistency of overparameterized neural networks. We then use this theory to derive Lipschitz-based generalization bounds for GANs. Our bounds show that penalizing the Lipschitz constant of the GAN loss can improve generalization. This result answers the long mystery of why the popular use of Lipschitz constraint for GANs often leads to great success, empirically without a solid theory. Finally but surprisingly, we show that, when using Dropout or spectral normalization, both truly deep neural networks and GANs can generalize well without the curse of dimensionality.

Key facts. Cited by 2 papers in the map (data through 14 Oct 2026). Builds on 52 of its 84 references that are in the map. First posted on arXiv 6 Apr 2021.

Builds on

52 of 84 references.

References · 52

  1. 2018A Style-Based Generator Architecture for Generative Adversarial Networkscited by 6,013
  2. 2019Analyzing and Improving the Image Quality of StyleGANcited by 3,588
  3. 2016Improved Techniques for Training GANscited by 2,970
  4. 2018Large Scale GAN Training for High Fidelity Natural Image Synthesiscited by 2,626
  5. 2018Neural Tangent Kernel: Convergence and Generalization in Neural Networkscited by 2,621
  6. 2017Improved Training of Wasserstein GANscited by 2,235
  7. 2018Spectral Normalization for Generative Adversarial Networkscited by 1,636
  8. 2020Training Generative Adversarial Networks with Limited Datacited by 1,186
  9. 2016Least Squares Generative Adversarial Networkscited by 1,001
  10. 2018Reconciling modern machine learning practice and the bias-variance trade-offcited by 927
  11. 2018Self-Attention Generative Adversarial Networkscited by 809
  12. 2018A Convergence Theory for Deep Learning via Over-Parameterizationcited by 787
38 more
  1. 2017Spectrally-normalized margin bounds for neural networkscited by 686
  2. 2018Which Training Methods for GANs do actually Converge?cited by 679
  3. 2018Gradient Descent Finds Global Minima of Deep Neural Networkscited by 574
  4. 2019Benign Overfitting in Linear Regressioncited by 549
  5. 2016f-GAN: Training Generative Neural Samplers using Variational Divergence Minimizationcited by 506
  6. 2017Towards Principled Methods for Training Generative Adversarial Networkscited by 464
  7. 2017Wasserstein Auto-Encoderscited by 373
  8. 2020Differentiable Augmentation for Data-Efficient GAN Trainingcited by 363
  9. 2014Breaking the Curse of Dimensionality with Convex Neural Networkscited by 331
  10. 2017A PAC-Bayesian Approach to Spectrally-Normalized Margin Bounds for Neural Networkscited by 314
  11. 2018Regularisation of Neural Networks by Enforcing Lipschitz Continuitycited by 314
  12. 2017Size-Independent Sample Complexity of Neural Networkscited by 290
  13. 2017Are GANs Created Equal? A Large-Scale Studycited by 267
  14. 2019Uniform convergence may be unable to explain generalization in deep learningcited by 224
  15. 2017Generalization and Equilibrium in Generative Adversarial Nets (GANs)cited by 166
  16. 2013Minimizing Finite Sums with the Stochastic Average Gradientcited by 164
  17. 2019Consistency Regularization for Generative Adversarial Networkscited by 136
  18. 2016Katyusha: The First Direct Acceleration of Stochastic Gradient Methodscited by 135
  19. 2017Stabilizing Training of Generative Adversarial Networks through Regularizationcited by 118
  20. 2017The Numerics of GANscited by 105
  21. 2020Training robust neural networks using Lipschitz boundscited by 99
  22. 2020Image Augmentations for GAN Trainingcited by 78
  23. 2017Gradient descent GAN optimization is locally stablecited by 67
  24. 2020Global Convergence of Deep Networks with One Wide Layer Followed by Pyramidal Topologycited by 66
  25. 2018On the Convergence and Robustness of Training GANs with Regularized Optimal Transportcited by 51
  26. 2017Many Paths to Equilibrium: GANs Do Not Need to Decrease a Divergence At Every Stepcited by 45
  27. 2019In Defense of Uniform Convergence: Generalization via derandomization with an application to interpolating predictorscited by 44
  28. 2021On the Proof of Global Convergence of Gradient Descent for Deep ReLU Networks with Linear Widthscited by 44
  29. 2020Regularization Matters: A Nonparametric Perspective on Overparametrized Neural Networkcited by 38
  30. 2017Loss-Sensitive Generative Adversarial Networks on Lipschitz Densitiescited by 29
  31. 2020Smoothness and Stability in GANscited by 27
  32. 2020Dropout: Explicit Forms and Capacity Controlcited by 27
  33. 2019On Stabilizing Generative Adversarial Training with Noisecited by 25
  34. 2020Learning disconnected manifolds: a no GANs landcited by 25
  35. 2019Lipschitz Generative Adversarial Netscited by 24
  36. 2019Generalization in Generative Adversarial Networks: A Novel Perspective from Privacy Protectioncited by 19
  37. 2020On Convergence and Generalization of Dropout Trainingcited by 19
  38. 2017On the Effectiveness of Least Squares Generative Adversarial Networkscited by 18

50 most cited of 52 All 52 (JSON)

Built on by

Cited by year

Citing papers per year: 2023: 1; 2024: 1.

2 papers.

  1. 2023PDE+: Enhancing Generalization via PDE with Adaptive Distributional Diffusioncited by 3
  2. 2024A Wasserstein perspective of Vanilla GANscited by 3