mlsanityData to 14 Oct 2026

Pros and Cons of GAN Evaluation Measures: New Developments

arXiv 2103.09396 ↗First posted 17 Mar 2021cs.LG cs.AI cs.CVcited by 123

Not on the map. Why?

Cited by 25 in Diffusion models, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

This work is an update of a previous paper on the same topic published a few years ago. With the dramatic progress in generative modeling, a suite of new quantitative and qualitative techniques to evaluate models has emerged. Although some measures such as Inception Score, Frechet Inception Distance, Precision-Recall, and Perceptual Path Length are relatively more popular, GAN evaluation is not a settled issue and there is still room for improvement. Here, I describe new dimensions that are becoming important in assessing models (e.g. bias and fairness) and discuss the connection between GAN evaluation and deepfakes. These are important areas of concern in the machine learning community today and progress in GAN evaluation can help mitigate them.

Key facts. Cited by 123 papers in the map (data through 14 Oct 2026). Builds on 64 of its 93 references that are in the map. First posted on arXiv 17 Mar 2021.

Builds on

64 of 93 references.

References · 64

  1. 2014Very Deep Convolutional Networks for Large-Scale Image Recognitioncited by 15,364
  2. 2018The Unreasonable Effectiveness of Deep Features as a Perceptual Metriccited by 11,177
  3. 2018A Style-Based Generator Architecture for Generative Adversarial Networkscited by 6,013
  4. 2021Zero-Shot Text-to-Image Generationcited by 4,288
  5. 2014Deep Learning Face Attributes in the Wildcited by 3,693
  6. 2019Analyzing and Improving the Image Quality of StyleGANcited by 3,588
  7. 2017Quo Vadis, Action Recognition? A New Model and the Kinetics Datasetcited by 3,270
  8. 2016Improved Techniques for Training GANscited by 2,970
  9. 2018Large Scale GAN Training for High Fidelity Natural Image Synthesiscited by 2,626
  10. 2019Generating Diverse High-Fidelity Images with VQ-VAE-2cited by 1,387
  11. 2015LSUN: Construction of a Large-scale Image Dataset using Deep Learning with Humans in the Loopcited by 1,355
  12. 2019Semantic Image Synthesis with Spatially-Adaptive Normalizationcited by 1,299
38 more
  1. 2018Demystifying MMD GANscited by 1,293
  2. 2019Improved Precision and Recall Metric for Assessing Generative Modelscited by 944
  3. 2016Conditional Image Generation with PixelCNN Decoderscited by 862
  4. 2019CNN-generated images are surprisingly easy to spot... for nowcited by 847
  5. 2017MoCoGAN: Decomposing Motion and Content for Video Generationcited by 524
  6. 2020Leveraging Frequency Analysis for Deep Fake Image Recognitioncited by 504
  7. 2020Reliable Fidelity and Diversity Metrics for Generative Modelscited by 434
  8. 2018Assessing Generative Models via Precision and Recallcited by 406
  9. 2015A note on the evaluation of generative modelscited by 292
  10. 2020Watch your Up-Convolution: CNN Based Generative Deep Neural Networks are Failing to Reproduce Spectral Distributionscited by 280
  11. 2020What makes fake images detectable? Understanding properties that generalizecited by 273
  12. 2017Are GANs Created Equal? A Large-Scale Studycited by 267
  13. 2021Generating Images with Sparse Representationscited by 266
  14. 2021Deep Generative Modelling: A Comparative Review of VAEs, GANs, Normalizing Flows, Energy-Based and Autoregressive Modelscited by 252
  15. 2019On the "steerability" of generative adversarial networkscited by 248
  16. 2020DeepFakes and Beyond: A Survey of Face Manipulation and Fake Detectioncited by 233
  17. 2018A Note on the Inception Scorecited by 226
  18. 2018Pros and Cons of GAN Evaluation Measurescited by 224
  19. 2015Deep Generative Image Models using a Laplacian Pyramid of Adversarial Networkscited by 214
  20. 2021How Faithful is your Synthetic Data? Sample-level Metrics for Evaluating and Auditing Generative Modelscited by 196
  21. 2018Fr'echet ChemNet Distance: A metric for generative models for molecules in drug discoverycited by 194
  22. 2019Effectively Unbiased FID and Inception Score and where to find themcited by 165
  23. 2018FairGAN: Fairness-aware Generative Adversarial Networkscited by 164
  24. 2018Lip Movements Generation at a Glancecited by 155
  25. 2019Seeing What a GAN Cannot Generatecited by 148
  26. 2019Fourier Spectrum Discrepancies in Deep Network Generated Imagescited by 139
  27. 2019Classification Accuracy Score for Conditional Generative Modelscited by 136
  28. 2021Generative Adversarial Transformerscited by 120
  29. 2020A Review on Deep Learning Techniques for Video Predictioncited by 114
  30. 2018Bias and Generalization in Deep Generative Models: An Empirical Studycited by 83
  31. 2018How good is my GAN?cited by 74
  32. 2018Pose Guided Human Video Generationcited by 67
  33. 2019HYPE: A Benchmark for Human eYe Perceptual Evaluation of Generative Modelscited by 59
  34. 2018Geometry Score: A Method For Comparing Generative Adversarial Networkscited by 57
  35. 2020A Non-Parametric Test to Detect Data-Copying in Generative Modelscited by 52
  36. 2020Generative Adversarial Networks for Image and Video Synthesis: Algorithms and Applicationscited by 48
  37. 2020Inclusive GAN: Improving Data and Minority Coverage in Generative Modelscited by 44
  38. 2019The Shape of Data: Intrinsic Distance for Data Distributionscited by 38

50 most cited of 64 All 64 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2021: 7; 2022: 29; 2023: 32; 2024: 28; 2025: 13; 2026: 14.

123 papers.

Sort:

By area:

  1. 2022A Survey on Generative Diffusion Modelcited by 240
  2. 2022The Role of ImageNet Classes in Fr'echet Inception Distancecited by 213
  3. 2023Exposing flaws of generative model evaluation metrics and their unfair treatment of diffusion modelscited by 205
  4. 2023A Categorical Archive of ChatGPT Failurescited by 168
  5. 2022Diffusion Probabilistic Models beat GANs on Medical Imagescited by 77
  6. 2023ImagenHub: Standardizing the evaluation of conditional image generation modelscited by 77
  7. 2021Unleashing Transformers: Parallel Token Prediction with Discrete Absorbing Diffusion for Fast High-Resolution Image Generation from Vector-Quantized Codescited by 71
  8. 2022Generated Faces in the Wild: Quantitative Comparison of Stable Diffusion, Midjourney and DALL-E 2cited by 55
  9. 2023LDM3D: Latent Diffusion Model for 3Dcited by 53
  10. 2024On the Content Bias in Fr'echet Video Distancecited by 52
  11. 2021Generating Synthetic Mixed-type Longitudinal Electronic Health Records for Artificial Intelligent Applicationscited by 47
  12. 2021Particle Cloud Generation with Message Passing Generative Adversarial Networkscited by 46
38 more
  1. 2022Turning the Tables: Biased, Imbalanced, Dynamic Tabular Datasets for ML Evaluationcited by 40
  2. 2022Machine Learning in Aerodynamic Shape Optimizationcited by 39
  3. 2022A Study on the Evaluation of Generative Modelscited by 35
  4. 2023Affective Faces for Goal-Driven Dyadic Communicationcited by 35
  5. 2023Qualitative Failures of Image Generation Models and Their Application in Detecting Deepfakescited by 31
  6. 2024On the Challenges and Opportunities in Generative AIcited by 31
  7. 2022Diagnosing and Fixing Manifold Overfitting in Deep Generative Modelscited by 28
  8. 2023Beyond Statistical Similarity: Rethinking Metrics for Deep Generative Models in Engineering Designcited by 23
  9. 2023Beware of diffusion models for synthesizing medical images -- A comparison with GANs in terms of memorizing brain MRI and chest x-ray imagescited by 23
  10. 2024Beyond FVD: Enhanced Evaluation Metrics for Video Generation Qualitycited by 22
  11. 2021Generative adversarial networks in time series: A survey and taxonomycited by 21
  12. 2024Similarity and Quality Metrics for MR Image-To-Image Translationcited by 20
  13. 2023Deep Generative Models for Detector Signature Simulation: A Taxonomic Reviewcited by 18
  14. 2024An Interpretable Evaluation of Entropy-based Novelty of Generative Modelscited by 17
  15. 2024Unveiling Differences in Generative Models: A Scalable Differential Clustering Approachcited by 16
  16. 2021Data synthesis and adversarial networks: A review and meta-analysis in cancer imagingcited by 15
  17. 2022medigan: a Python library of pretrained generative models for medical image synthesiscited by 14
  18. 2024Image Neural Field Diffusion Modelscited by 13
  19. 2022DALLE-URBAN: Capturing the urban design expertise of large text to image transformerscited by 12
  20. 2025Position: All Current Generative Fidelity and Diversity Metrics are Flawedcited by 12
  21. 2022Assessing the ability of generative adversarial networks to learn canonical medical image statisticscited by 11
  22. 2022Human Image Generation: A Comprehensive Surveycited by 11
  23. 2022Exploring Vision Transformers as Diffusion Learnerscited by 10
  24. 2023Structural Cycle GAN for Virtual Immunohistochemistry Staining of Gland Markers in the Coloncited by 10
  25. 2025Survey on the Evaluation of Generative Models in Musiccited by 10
  26. 2023This Intestine Does Not Exist: Multiscale Residual Variational Autoencoder for Realistic Wireless Capsule Endoscopy Image Generationcited by 9
  27. 2023Translation-Enhanced Multilingual Text-to-Image Generationcited by 9
  28. 2023Pre- to Post-Contrast Breast MRI Synthesis for Enhanced Tumour Segmentationcited by 9
  29. 2024Unconditional Latent Diffusion Models Memorize Patient Imaging Data: Implications for Openly Sharing Synthetic Datacited by 9
  30. 2024Slight Corruption in Pre-training Data Makes Better Diffusion Modelscited by 9
  31. 2024Geometry Fidelity for Spherical Imagescited by 9
  32. 2024Refereeing the Referees: Evaluating Two-Sample Tests for Validating Generators in Precision Sciencescited by 9
  33. 2022Generalization Metrics for Practical Quantum Advantage in Generative Modelscited by 8
  34. 2022Approaching Peak Ground Truthcited by 8
  35. 2022Differentially Private Generative Adversarial Networks with Model Inversioncited by 7
  36. 2022CycleGAN for Undamaged-to-Damaged Domain Translation for Structural Health Monitoring and Damage Detectioncited by 7
  37. 2022Instability and Local Minima in GAN Training with Kernel Discriminatorscited by 7
  38. 2023Assessing the capacity of a denoising diffusion probabilistic model to reproduce spatial contextcited by 7

50 most cited of 123 All 123 (JSON)