mlsanityData to 7 Oct 2026

A Neural Tangent Kernel Perspective of GANs

arXiv 2106.05566 ↗First posted 10 Jun 2021cs.LG cs.NE stat.MLcited by 21

Not on the map. Why?

Abstract (arXiv)

We propose a novel theoretical framework of analysis for Generative Adversarial Networks (GANs). We reveal a fundamental flaw of previous analyses which, by incorrectly modeling GANs' training scheme, are subject to ill-defined discriminator gradients. We overcome this issue which impedes a principled study of GAN training, solving it within our framework by taking into account the discriminator's architecture. To this end, we leverage the theory of infinite-width neural networks for the discriminator via its Neural Tangent Kernel. We characterize the trained discriminator for a wide range of losses and establish general differentiability properties of the network. From this, we derive new insights about the convergence of the generated distribution, advancing our understanding of GANs' training dynamics. We empirically corroborate these results via an analysis toolkit based on our framework, unveiling intuitions that are consistent with GAN practice.

Key facts. Cited by 21 papers in the map (data through 7 Oct 2026). Builds on 61 of its 119 references that are in the map. First posted on arXiv 10 Jun 2021.

Builds on

61 of 119 references.

References · 61

  1. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  2. 2018A Style-Based Generator Architecture for Generative Adversarial Networkscited by 5,986
  3. 2014Deep Learning Face Attributes in the Wildcited by 3,679
  4. 2019Analyzing and Improving the Image Quality of StyleGANcited by 3,580
  5. 2018Large Scale GAN Training for High Fidelity Natural Image Synthesiscited by 2,609
  6. 2018Neural Tangent Kernel: Convergence and Generalization in Neural Networkscited by 2,597
  7. 2020Fourier Features Let Networks Learn High Frequency Functions in Low Dimensional Domainscited by 2,170
  8. 2015Unsupervised Representation Learning with Deep Convolutional Generative Adversarial Networkscited by 2,048
  9. 2016Least Squares Generative Adversarial Networkscited by 1,000
  10. 2018A Convergence Theory for Deep Learning via Over-Parameterizationcited by 784
  11. 2019Wide Neural Networks of Any Depth Evolve as Linear Models Under Gradient Descentcited by 732
  12. 2018Which Training Methods for GANs do actually Converge?cited by 672
38 more
  1. 2017Deep Neural Networks as Gaussian Processescited by 617
  2. 2019On Exact Computation with an Infinitely Wide Neural Netcited by 572
  3. 2016Stein Variational Gradient Descent: A General Purpose Bayesian Inference Algorithmcited by 498
  4. 2017Towards Principled Methods for Training Generative Adversarial Networkscited by 460
  5. 2016Kernel Mean Embedding of Distributions: A Review and Beyondcited by 360
  6. 2016NIPS 2016 Tutorial: Generative Adversarial Networkscited by 284
  7. 2017Are GANs Created Equal? A Large-Scale Studycited by 267
  8. 2017MMD GAN: Towards Deeper Understanding of Moment Matching Networkcited by 251
  9. 2017Geometric GANcited by 232
  10. 2020DeepFakes and Beyond: A Survey of Face Manipulation and Fake Detectioncited by 231
  11. 2016Unrolled Generative Adversarial Networkscited by 208
  12. 2017VEEGAN: Reducing Mode Collapse in GANs using Implicit Variational Learningcited by 199
  13. 2020Feature Learning in Infinite-Width Neural Networkscited by 182
  14. 2019On the Inductive Bias of Neural Tangent Kernelscited by 177
  15. 2017Generalization and Equilibrium in Generative Adversarial Nets (GANs)cited by 165
  16. 2020Finite Versus Infinite Neural Networks: an Empirical Studycited by 156
  17. 2019Neural Tangents: Fast and Easy Infinite Neural Networks in Pythoncited by 149
  18. 2020Frequency Bias in Neural Networks for Input of Non-Uniform Densitycited by 140
  19. 2020Tensor Programs II: Neural Tangent Kernel for Any Architecturecited by 139
  20. 2020On the linearity of large non-linear models: when and why the tangent kernel is constantcited by 129
  21. 2019Maximum Mean Discrepancy Gradient Flowcited by 123
  22. 2020Infinite attention: NNGP and NTK for deep attention networkscited by 117
  23. 2017The Numerics of GANscited by 104
  24. 2019Harnessing the Power of Infinitely Wide Deep Nets on Small-data Taskscited by 87
  25. 2020Deep Equals Shallow for ReLU Networks in Kernel Regimescited by 83
  26. 2019On the geometry of Stein variational gradient descentcited by 82
  27. 2020Spectra of the Conjugate Kernel and Neural Tangent Kernel for linear-width neural networkscited by 76
  28. 2020Deep Neural Tangent Kernel and Laplace Kernel Have the Same RKHScited by 74
  29. 2016Amortised MAP Inference for Image Super-resolutioncited by 68
  30. 2019A Fine-Grained Spectral Perspective on Neural Networkscited by 68
  31. 2020The Recurrent Neural Tangent Kernelcited by 67
  32. 2017Gradient descent GAN optimization is locally stablecited by 65
  33. 2020Why Do Deep Residual Networks Generalize Better than Deep Feedforward Networks? -- A Neural Tangent Kernel Perspectivecited by 60
  34. 2020Generative Adversarial Networks for Image and Video Synthesis: Algorithms and Applicationscited by 48
  35. 2020On the infinite width limit of neural networks with a standard parameterizationcited by 38
  36. 2019A type of generalization error induced by initialization in deep neural networkscited by 33
  37. 2019Neural tangent kernels, transportation mappings, and universal approximationcited by 30
  38. 2017Approximation and Convergence Properties of Generative Adversarial Learningcited by 28

50 most cited of 61 All 61 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 6; 2023: 8; 2024: 5; 2025: 1; 2026: 1.

21 papers.

Sort:

By area:

  1. 2022Fast Finite Width Neural Tangent Kernelcited by 62
  2. 2022A Fast, Well-Founded Approximation to the Empirical Neural Tangent Kernelcited by 30
  3. 2023Unifying GANs and Score-Based Diffusion as Generative Particle Modelscited by 21
  4. 2024Emergent Equivariance in Deep Ensemblescited by 18
  5. 2024Deep MMD Gradient Flow without adversarial trainingcited by 18
  6. 2023Differentially Private Neural Tangent Kernels for Privacy-Preserving Data Generationcited by 15
  7. 2023MonoFlow: Rethinking Divergence GANs via the Perspective of Wasserstein Gradient Flowscited by 12
  8. 2024Peer-to-Peer Learning Dynamics of Wide Neural Networkscited by 8
  9. 2022Instability and Local Minima in GAN Training with Kernel Discriminatorscited by 7
  10. 2023Federated Learning on Virtual Heterogeneous Data with Local-global Distillationcited by 6
  11. 2023A Theory of Unsupervised Speech Recognitioncited by 4
  12. 2024Equivariant Neural Tangent Kernelscited by 4
9 more
  1. 2024On the Convergence Analysis of Over-Parameterized Variational Autoencoders: A Neural Tangent Kernel Perspectivecited by 3
  2. 2025Finite-Width Neural Tangent Kernels from Feynman Diagramscited by 3
  3. 2022Single-level Adversarial Data Synthesis based on Neural Tangent Kernelscited by 2
  4. 2023Data Interpolants -- That's What Discriminators in Higher-order Gradient-regularized GANs Arecited by 2
  5. 2022There is a Singularity in the Loss Landscapecited by 1
  6. 2022Effective Dynamics of Generative Adversarial Networkscited by 1
  7. 2023Controlled Descent Trainingcited by 0
  8. 2023Insights into Closed-form IPM-GAN Discriminator Guidance for Diffusion Modelingcited by 0
  9. 2026Refresh or Realize? Compute Allocation in Drifting Modelscited by 0