mlsanityData to 14 Oct 2026

A Study on the Evaluation of Generative Models

arXiv 2206.10935 ↗First posted 22 Jun 2022cs.LG cs.CVcited by 35

On the map: Diffusion models → Diffusion foundations and sampling

Abstract (arXiv)

Implicit generative models, which do not return likelihood values, such as generative adversarial networks and diffusion models, have become prevalent in recent years. While it is true that these models have shown remarkable results, evaluating their performance is challenging. This issue is of vital importance to push research forward and identify meaningful gains from random noise. Currently, heuristic metrics such as the Inception score (IS) and Frechet Inception Distance (FID) are the most common evaluation metrics, but what they measure is not entirely clear. Additionally, there are questions regarding how meaningful their score actually is. In this work, we study the evaluation metrics of generative models by generating a high-quality synthetic dataset on which we can estimate classical metrics for comparison. Our study shows that while FID and IS do correlate to several f-divergences, their ranking of close models can vary considerably making them problematic when used for fain-grained comparison. We further used this experimental setting to study which evaluation metric best correlates with our probabilistic metrics. Lastly, we look into the base features used for metrics such as FID.

Key facts. Cited by 35 papers in the map (data through 14 Oct 2026). Placed in Diffusion foundations and sampling, one of 14 directions in Diffusion models. Builds on 22 of its 32 references that are in the map. First posted on arXiv 22 Jun 2022.

Builds on

22 of 32 references.

Closest ancestors · 1

Same direction, newest first

  1. 2021Improved Denoising Diffusion Probabilistic Models18 Feb

Other references · 21

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  2. 2022Hierarchical Text-Conditional Image Generation with CLIP Latentscited by 6,585
  3. 2014Going Deeper with Convolutionscited by 4,241
  4. 2014Deep Learning Face Attributes in the Wildcited by 3,693
  5. 2019Analyzing and Improving the Image Quality of StyleGANcited by 3,588
  6. 2016Improved Techniques for Training GANscited by 2,970
  7. 2015LSUN: Construction of a Large-scale Image Dataset using Deep Learning with Humans in the Loopcited by 1,355
  8. 2018Demystifying MMD GANscited by 1,293
  9. 2020Training Generative Adversarial Networks with Limited Datacited by 1,186
  10. 2020HiFi-GAN: Generative Adversarial Networks for Efficient and High Fidelity Speech Synthesiscited by 1,154
  11. 2019StarGAN v2: Diverse Image Synthesis for Multiple Domainscited by 1,098
  12. 2015Importance Weighted Autoencoderscited by 456
9 more
  1. 2020Very Deep VAEs Generalize Autoregressive Models and Can Outperform Them on Imagescited by 264
  2. 2021Deep Generative Modelling: A Comparative Review of VAEs, GANs, Normalizing Flows, Energy-Based and Autoregressive Modelscited by 252
  3. 2018A Note on the Inception Scorecited by 226
  4. 2019Effectively Unbiased FID and Inception Score and where to find themcited by 165
  5. 2017PixelSNAIL: An Improved Autoregressive Generative Modelcited by 137
  6. 2021Pros and Cons of GAN Evaluation Measures: New Developmentscited by 123
  7. 2018An empirical study on evaluation metrics of generative adversarial networkscited by 81
  8. 2021Generating images from caption and vice versa via CLIP-Guided Generative Latent Space Searchcited by 57
  9. 2021On Aliased Resizing and Surprising Subtleties in GAN Evaluationcited by 51

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 11; 2024: 18; 2025: 4; 2026: 2.

35 papers.

Sort:

By area:

  1. 2023PixArt-α: Fast Training of Diffusion Transformer for Photorealistic Text-to-Image Synthesiscited by 864
  2. 2023Exposing flaws of generative model evaluation metrics and their unfair treatment of diffusion modelscited by 205
  3. 2023Data-centric Artificial Intelligence: A Surveycited by 203
  4. 2023UFOGen: You Forward Once Large Scale Text-to-Image Generation via Diffusion GANscited by 166
  5. 2024xDiT: an Inference Engine for Diffusion Transformers (DiTs) with Massive Parallelismcited by 56
  6. 2024LAPTOP-Diff: Layer Pruning and Normalized Distillation for Compressing Diffusion Modelscited by 53
  7. 2023Diffusion Illusions: Hiding Images in Plain Sightcited by 26
  8. 2023KOALA: Empirical Lessons Toward Memory-Efficient and Fast Diffusion Models for Text-to-Image Synthesiscited by 18
  9. 2023Deep Generative Models for Detector Signature Simulation: A Taxonomic Reviewcited by 18
  10. 2025Generative Models in Decision Making: A Surveycited by 15
  11. 2024SCott: Accelerating Diffusion Models with Stochastic Consistency Distillationcited by 14
  12. 2024A Practical Guide to Sample-based Statistical Distances for Evaluating Generative Models in Sciencecited by 14
23 more
  1. 2024Slight Corruption in Pre-training Data Makes Better Diffusion Modelscited by 9
  2. 2025Reinforcement Learning for Large Model: A Surveycited by 9
  3. 2023Revisiting the Evaluation of Image Synthesis with GANscited by 7
  4. 2023Navigating the Synthetic Realm: Harnessing Diffusion-based Models for Laparoscopic Text-to-Image Generationcited by 7
  5. 2023On the Noise Scheduling for Generating Plausible Designs with Diffusion Modelscited by 6
  6. 2024Surgical Text-to-Image Generationcited by 6
  7. 2023GMValuator: Similarity-based Data Valuation for Generative Modelscited by 5
  8. 2024Fr'echet Denoised Distance: Enhancing Plausibility Evaluation for Generated Designs with Denoising Autoencodercited by 3
  9. 2024Segmentation-Free Guidance for Text-to-Image Diffusion Modelscited by 3
  10. 2024Semantic Approach to Quantifying the Consistency of Diffusion Model Image Generationcited by 2
  11. 2024DiffFinger: Advancing Synthetic Fingerprint Generation through Denoising Diffusion Probabilistic Modelscited by 2
  12. 2024Survey on Visual Signal Coding and Processing with Generative Models: Technologies, Standards and Optimizationcited by 2
  13. 2024ShapeWords: Guiding Text-to-Image Synthesis with 3D Shape-Aware Promptscited by 2
  14. 2025Quantifying the uncertainty of model-based synthetic image quality metricscited by 2
  15. 2025Beyond Real Data: Synthetic Data through the Lens of Regularizationcited by 2
  16. 2024Analyzing the Feature Extractor Networks for Face Image Synthesiscited by 1
  17. 2024Unleashing the Potential of Synthetic Images: A Study on Histopathology Image Classificationcited by 1
  18. 2024Improving Android Malware Detection Through Data Augmentation Using Wasserstein Generative Adversarial Networkscited by 0
  19. 2024Bird Eye-View to Street-View: A Surveycited by 0
  20. 2024IQA-Adapter: Exploring Knowledge Transfer from Image Quality Assessment to Diffusion-based Generative Modelscited by 0
  21. 2024Generating floorplans for various building functionalities via latent diffusion modelcited by 0
  22. 2026Component-Aware Sketch-to-Image Generation Using Self-Attention Encoding and Coordinate-Preserving Fusioncited by 0
  23. 2026RA-ClipScore: Making Generative Model Evaluation More Interpretablecited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2023Beyond Statistical Similarity: Rethinking Metrics for Deep Generative Models in Engineering Design
  2. 2026The FID Lottery: Quantifying Hidden Randomness in Generative-Model Evaluation
  3. 2024Evaluation Metrics and Methods for Generative Models in the Wireless PHY Layer
  4. 2024An analysis of the noise schedule for score-based generative models
  5. 2023The Score-Difference Flow for Implicit Generative Modeling
  6. 2024GenAI Arena: An Open Evaluation Platform for Generative Models
  7. 2025On the Anisotropy of Score-Based Generative Models
  8. 2024Image Generation Diversity Issues and How to Tame Them