# On the Discrimination-Generalization Tradeoff in GANs

arXiv: https://arxiv.org/abs/1711.02771 · HTML: https://mlsanity.com/paper/1711.02771/

## Key facts

- Cited by 48 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 7 Nov 2017.

## Built on by (most cited first, 30 of 48)

- [Exploring Sequence Feature Alignment for Domain Adaptive Detection Transformers](https://mlsanity.com/paper/2107.12636/) (2021)
- [KALE Flow: A Relaxed KL Gradient Flow for Probabilities with Disjoint Support](https://mlsanity.com/paper/2106.08929/) (2021)
- [An error analysis of generative adversarial networks for learning distributions](https://mlsanity.com/paper/2105.13010/) (2021)
- [MaGNET: Uniform Sampling from Deep Generative Network Manifolds Without Retraining](https://mlsanity.com/paper/2110.08009/) (2021)
- [A Survey on Statistical Theory of Deep Learning: Approximation, Training Dynamics, and Generative Models](https://mlsanity.com/paper/2401.07187/) (2024)
- [Deep MMD Gradient Flow without adversarial training](https://mlsanity.com/paper/2405.06780/) (2024)
- [On the Privacy Properties of GAN-generated Samples](https://mlsanity.com/paper/2206.01349/) (2022)
- [What is a Good Metric to Study Generalization of Minimax Learners?](https://mlsanity.com/paper/2206.04502/) (2022)
- [Statistical Guarantees for Variational Autoencoders using PAC-Bayesian Theory](https://mlsanity.com/paper/2310.04935/) (2023)
- [Smooth p-Wasserstein Distance: Structure, Empirical Approximation, and Statistical Applications](https://mlsanity.com/paper/2101.04039/) (2021)
- [Non-Asymptotic Performance Guarantees for Neural Estimation of mathsff-Divergences](https://mlsanity.com/paper/2103.06923/) (2021)
- [PAC-Bayesian Generalization Bounds for Adversarial Generative Models](https://mlsanity.com/paper/2302.08942/) (2023)
- [On Deep Generative Models for Approximation and Estimation of Distributions on Manifolds](https://mlsanity.com/paper/2302.13183/) (2023)
- [A likelihood approach to nonparametric estimation of a singular distribution using deep generative models](https://mlsanity.com/paper/2105.04046/) (2021)
- [Statistical Complexity of Quantum Learning](https://mlsanity.com/paper/2309.11617/) (2023)
- [A Communication-efficient Algorithm with Linear Convergence for Federated Minimax Learning](https://mlsanity.com/paper/2206.01132/) (2022)
- [The Representation Jensen-Shannon Divergence](https://mlsanity.com/paper/2305.16446/) (2023)
- [Generalization Error of GAN from the Discriminator's Perspective](https://mlsanity.com/paper/2107.03633/) (2021)
- [Self-Ensembling GAN for Cross-Domain Semantic Segmentation](https://mlsanity.com/paper/2112.07999/) (2021)
- [A Mathematical Framework for Learning Probability Distributions](https://mlsanity.com/paper/2212.11481/) (2022)
- [Imitation Learning from Pixel-Level Demonstrations by HashReward](https://mlsanity.com/paper/1909.03773/) (2019)
- [Minimax Optimality (Probably) Doesn't Imply Distribution Learning for GANs](https://mlsanity.com/paper/2201.07206/) (2022)
- [Quantum Neural Estimation of Entropies](https://mlsanity.com/paper/2307.01171/) (2023)
- [α-GAN: Convergence and Estimation Guarantees](https://mlsanity.com/paper/2205.06393/) (2022)
- [Neural network-based CUSUM for online change-point detection](https://mlsanity.com/paper/2210.17312/) (2022)
- [Addressing GAN Training Instabilities via Tunable Classification Losses](https://mlsanity.com/paper/2310.18291/) (2023)
- [Stability and Generalization in Free Adversarial Training](https://mlsanity.com/paper/2404.08980/) (2024)
- [On Value Discrepancy of Imitation Learning](https://mlsanity.com/paper/1911.07027/) (2019)
- [Non-Asymptotic Error Bounds for Bidirectional GANs](https://mlsanity.com/paper/2110.12319/) (2021)
- [Information Processing Equalities and the Information-Risk Bridge](https://mlsanity.com/paper/2207.11987/) (2022)

## Abstract (arXiv)

Generative adversarial training can be generally understood as minimizing certain moment matching loss defined by a set of discriminator functions, typically neural networks. The discriminator set should be large enough to be able to uniquely identify the true distribution (discriminative), and also be small enough to go beyond memorizing samples (generalizable). In this paper, we show that a discriminator set is guaranteed to be discriminative whenever its linear span is dense in the set of bounded continuous functions. This is a very mild condition satisfied even by neural networks with a single neuron. Further, we develop generalization bounds between the learned distribution and true distribution under different evaluation metrics. When evaluated with neural distance, our bounds show that generalization is guaranteed as long as the discriminator set is small enough, regardless of the size of the generator or hypothesis set. When evaluated with KL divergence, our bound provides an explanation on the counter-intuitive behaviors of testing likelihood in GAN training. Our analysis sheds lights on understanding the practical performance of GANs.