# An empirical study on evaluation metrics of generative adversarial networks

arXiv: https://arxiv.org/abs/1806.07755 · HTML: https://mlsanity.com/paper/1806.07755/

## Key facts

- Cited by 81 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 19 Jun 2018.

## Built on by (most cited first, 30 of 81)

- [GAN Inversion: A Survey](https://mlsanity.com/paper/2101.05278/) (2021)
- [Rethinking FID: Towards a Better Evaluation Metric for Image Generation](https://mlsanity.com/paper/2401.09603/) (2023)
- [The Role of ImageNet Classes in Fr'echet Inception Distance](https://mlsanity.com/paper/2203.06026/) (2022)
- [On Memorization in Probabilistic Deep Generative Models](https://mlsanity.com/paper/2106.03216/) (2021)
- [On Evaluation Metrics for Graph Generative Models](https://mlsanity.com/paper/2201.09871/) (2022)
- [Membership Inference on Text-to-Image Diffusion Models via Conditional Likelihood Discrepancy](https://mlsanity.com/paper/2405.14800/) (2024)
- [A Study on the Evaluation of Generative Models](https://mlsanity.com/paper/2206.10935/) (2022)
- [When do GANs replicate? On the choice of dataset size](https://mlsanity.com/paper/2202.11765/) (2022)
- [Facing the Elephant in the Room: Visual Prompt Tuning or Full Finetuning?](https://mlsanity.com/paper/2401.12902/) (2024)
- [Feature Likelihood Divergence: Evaluating the Generalization of Generative Models Using Samples](https://mlsanity.com/paper/2302.04440/) (2023)
- [Ultra-High-Resolution Detector Simulation with Intra-Event Aware GAN and Self-Supervised Relational Reasoning](https://mlsanity.com/paper/2303.08046/) (2023)
- [Flow Matching for Medical Image Synthesis: Bridging the Gap Between Speed and Quality](https://mlsanity.com/paper/2503.00266/) (2025)
- [Privacy-preserving Spatiotemporal Scenario Generation of Renewable Energies: A Federated Deep Generative Learning Approach](https://mlsanity.com/paper/2107.07738/) (2021)
- [A deep generative model for probabilistic energy forecasting in power systems: normalizing flows](https://mlsanity.com/paper/2106.09370/) (2021)
- [Synthetic ID Card Image Generation for Improving Presentation Attack Detection](https://mlsanity.com/paper/2211.00098/) (2022)
- [GUIGAN: Learning to Generate GUI Designs Using Generative Adversarial Networks](https://mlsanity.com/paper/2101.09978/) (2021)
- [Implicit Generative Copulas](https://mlsanity.com/paper/2109.14567/) (2021)
- [A Practical Guide to Sample-based Statistical Distances for Evaluating Generative Models in Science](https://mlsanity.com/paper/2403.12636/) (2024)
- [Efficient Conditional GAN Transfer with Knowledge Propagation across Classes](https://mlsanity.com/paper/2102.06696/) (2021)
- [Generating Physically-Consistent Satellite Imagery for Climate Visualizations](https://mlsanity.com/paper/2104.04785/) (2021)
- [Assessing a Single Image in Reference-Guided Image Synthesis](https://mlsanity.com/paper/2112.04163/) (2021)
- [Position: All Current Generative Fidelity and Diversity Metrics are Flawed](https://mlsanity.com/paper/2505.22450/) (2025)
- [All that structure matches does not glitter](https://mlsanity.com/paper/2509.12178/) (2025)
- [Generative Adversarial Networks (GANs) in Networking: A Comprehensive Survey & Evaluation](https://mlsanity.com/paper/2105.04184/) (2021)
- [A Deep-Learning Intelligent System Incorporating Data Augmentation for Short-Term Voltage Stability Assessment of Power Systems](https://mlsanity.com/paper/2112.03265/) (2021)
- [On the Robustness of Quality Measures for GANs](https://mlsanity.com/paper/2201.13019/) (2022)
- [Probabilistic Precision and Recall Towards Reliable Evaluation of Generative Models](https://mlsanity.com/paper/2309.01590/) (2023)
- [MVDD: Multi-View Depth Diffusion Models](https://mlsanity.com/paper/2312.04875/) (2023)
- [Data-Copying in Generative Models: A Formal Framework](https://mlsanity.com/paper/2302.13181/) (2023)
- [DiffECG: A Versatile Probabilistic Diffusion Model for ECG Signals Synthesis](https://mlsanity.com/paper/2306.01875/) (2023)

## Abstract (arXiv)

Evaluating generative adversarial networks (GANs) is inherently challenging. In this paper, we revisit several representative sample-based evaluation metrics for GANs, and address the problem of how to evaluate the evaluation metrics. We start with a few necessary conditions for metrics to produce meaningful scores, such as distinguishing real from generated samples, identifying mode dropping and mode collapsing, and detecting overfitting. With a series of carefully designed experiments, we comprehensively investigate existing sample-based metrics and identify their strengths and limitations in practical settings. Based on these results, we observe that kernel Maximum Mean Discrepancy (MMD) and the 1-Nearest-Neighbor (1-NN) two-sample test seem to satisfy most of the desirable properties, provided that the distances between samples are computed in a suitable feature space. Our experiments also unveil interesting properties about the behavior of several popular GAN models, such as whether they are memorizing training samples, and how far they are from learning the target distribution.