# Towards GAN Benchmarks Which Require Generalization

arXiv: https://arxiv.org/abs/2001.03653 · HTML: https://mlsanity.com/paper/2001.03653/

## Key facts

- Cited by 31 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 10 Jan 2020.

## Built on by (most cited first, 30 of 31)

- [Diffusion Art or Digital Forgery? Investigating Data Replication in Diffusion Models](https://mlsanity.com/paper/2212.03860/) (2022)
- [Deep Generative Modelling: A Comparative Review of VAEs, GANs, Normalizing Flows, Energy-Based and Autoregressive Models](https://mlsanity.com/paper/2103.04922/) (2021)
- [Pros and Cons of GAN Evaluation Measures: New Developments](https://mlsanity.com/paper/2103.09396/) (2021)
- [Regularizing Generative Adversarial Networks under Limited Data](https://mlsanity.com/paper/2104.03310/) (2021)
- [Membership Inference Attacks against Synthetic Data through Overfitting Detection](https://mlsanity.com/paper/2302.12580/) (2023)
- [On Memorization in Probabilistic Deep Generative Models](https://mlsanity.com/paper/2106.03216/) (2021)
- [Can Push-forward Generative Models Fit Multimodal Distributions?](https://mlsanity.com/paper/2206.14476/) (2022)
- [Feature Likelihood Divergence: Evaluating the Generalization of Generative Models Using Samples](https://mlsanity.com/paper/2302.04440/) (2023)
- [On Training Sample Memorization: Lessons from Benchmarking Generative Modeling with a Large-scale Competition](https://mlsanity.com/paper/2106.03062/) (2021)
- [KD-DLGAN: Data Limited Image Generation via Knowledge Distillation](https://mlsanity.com/paper/2303.17158/) (2023)
- [Understanding Overparameterization in Generative Adversarial Networks](https://mlsanity.com/paper/2104.05605/) (2021)
- [GenCo: Generative Co-training for Generative Adversarial Networks with Limited Data](https://mlsanity.com/paper/2110.01254/) (2021)
- [Survey on the Evaluation of Generative Models in Music](https://mlsanity.com/paper/2506.05104/) (2025)
- [Generalization Error of GAN from the Discriminator's Perspective](https://mlsanity.com/paper/2107.03633/) (2021)
- [Generalization Metrics for Practical Quantum Advantage in Generative Models](https://mlsanity.com/paper/2201.08770/) (2022)
- [A Mathematical Framework for Learning Probability Distributions](https://mlsanity.com/paper/2212.11481/) (2022)
- [An Inversion-based Measure of Memorization for Diffusion Models](https://mlsanity.com/paper/2405.05846/) (2024)
- [Reducing Training Sample Memorization in GANs by Training with Memorization Rejection](https://mlsanity.com/paper/2210.12231/) (2022)
- [Optimal 1-Wasserstein Distance for WGANs](https://mlsanity.com/paper/2201.02824/) (2022)
- [Statistically Optimal Generative Modeling with Maximum Deviation from the Empirical Distribution](https://mlsanity.com/paper/2307.16422/) (2023)
- [Are conditional GANs explicitly conditional?](https://mlsanity.com/paper/2106.15011/) (2021)
- [Understanding Generalization in Diffusion Distillation via Probability Flow Distance](https://mlsanity.com/paper/2505.20123/) (2025)
- [Compositional Fine-Grained Low-Shot Learning](https://mlsanity.com/paper/2105.10438/) (2021)
- [Intriguing Property and Counterfactual Explanation of GAN for Remote Sensing Image Generation](https://mlsanity.com/paper/2303.05240/) (2023)
- [Magmaw: Modality-Agnostic Adversarial Attacks on Machine Learning-Based Wireless Communication Systems](https://mlsanity.com/paper/2311.00207/) (2023)
- [Detecting Generative Parroting through Overfitting Masked Autoencoders](https://mlsanity.com/paper/2403.19050/) (2024)
- [MS3D: A RG Flow-Based Regularization for GAN Training with Limited Data](https://mlsanity.com/paper/2408.11135/) (2024)
- [Improving few-shot learning-based protein engineering with evolutionary sampling](https://mlsanity.com/paper/2305.15441/) (2023)
- [On depth prediction for autonomous driving using self-supervised learning](https://mlsanity.com/paper/2403.06194/) (2024)
- [Style Quantization for Data-Efficient GAN Training](https://mlsanity.com/paper/2503.24282/) (2025)

## Abstract (arXiv)

For many evaluation metrics commonly used as benchmarks for unconditional image generation, trivially memorizing the training set attains a better score than models which are considered state-of-the-art; we consider this problematic. We clarify a necessary condition for an evaluation metric not to behave this way: estimating the function must require a large sample from the model. In search of such a metric, we turn to neural network divergences (NNDs), which are defined in terms of a neural network trained to distinguish between distributions. The resulting benchmarks cannot be "won" by training set memorization, while still being perceptually correlated and computable only from samples. We survey past work on using NNDs for evaluation and implement an example black-box metric based on these ideas. Through experimental validation we show that it can effectively measure diversity, sample quality, and generalization.