mlsanityData to 7 Oct 2026

Rethinking FID: Towards a Better Evaluation Metric for Image Generation

arXiv 2401.09603 ↗First posted 30 Nov 2023cs.CVcited by 303

Not on the map. Why?

Cited by 119 in Diffusion models, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

As with many machine learning problems, the progress of image generation methods hinges on good evaluation metrics. One of the most popular is the Frechet Inception Distance (FID). FID estimates the distance between a distribution of Inception-v3 features of real images, and those of images generated by the algorithm. We highlight important drawbacks of FID: Inception's poor representation of the rich and varied content generated by modern text-to-image models, incorrect normality assumptions, and poor sample complexity. We call for a reevaluation of FID's use as the primary quality metric for generated images. We empirically demonstrate that FID contradicts human raters, it does not reflect gradual improvement of iterative text-to-image models, it does not capture distortion levels, and that it produces inconsistent results when varying the sample size. We also propose an alternative new metric, CMMD, based on richer CLIP embeddings and the maximum mean discrepancy distance with the Gaussian RBF kernel. It is an unbiased estimator that does not make any assumptions on the probability distribution of the embeddings and is sample efficient. Through extensive experiments and analysis, we demonstrate that FID-based evaluations of text-to-image models may be unreliable, and that CMMD offers a more robust and reliable assessment of image quality.

Key facts. Cited by 303 papers in the map (data through 7 Oct 2026). Builds on 14 of its 26 references that are in the map. First posted on arXiv 30 Nov 2023.

Builds on

14 of 26 references.

References · 14

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  2. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,694
  3. 2014Microsoft COCO: Common Objects in Contextcited by 17,498
  4. 2022Hierarchical Text-Conditional Image Generation with CLIP Latentscited by 6,539
  5. 2022Photorealistic Text-to-Image Diffusion Models with Deep Language Understandingcited by 6,322
  6. 2018A Style-Based Generator Architecture for Generative Adversarial Networkscited by 5,986
  7. 2015Rethinking the Inception Architecture for Computer Visioncited by 5,451
  8. 2020Taming Transformers for High-Resolution Image Synthesiscited by 3,229
  9. 2016Improved Techniques for Training GANscited by 2,938
  10. 2022Scaling Autoregressive Models for Content-Rich Text-to-Image Generationcited by 1,287
  11. 2023Muse: Text-To-Image Generation via Masked Generative Transformerscited by 640
  12. 2018A Note on the Inception Scorecited by 225
2 more
  1. 2019Effectively Unbiased FID and Inception Score and where to find themcited by 161
  2. 2018An empirical study on evaluation metrics of generative adversarial networkscited by 81

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 2; 2024: 73; 2025: 130; 2026: 98.

303 papers.

Sort:

By area:

  1. 2025Hunyuan3D 2.0: Scaling Diffusion Models for High Resolution Textured 3D Assets Generationcited by 342
  2. 2024Deep Compression Autoencoder for Efficient High-Resolution Diffusion Modelscited by 209
  3. 2024BrushNet: A Plug-and-Play Image Inpainting Model with Decomposed Dual-Branch Diffusioncited by 200
  4. 2024DOCCI: Descriptions of Connected and Contrasting Imagescited by 130
  5. 2024Self-Rectifying Diffusion Sampling with Perturbed-Attention Guidancecited by 108
  6. 2025MEt3R: Measuring Multi-View Consistency in Generated Imagescited by 91
  7. 2025What matters for Representation Alignment: Global Information or Spatial Structure?cited by 76
  8. 2024DiffH2O: Diffusion-Based Synthesis of Hand-Object Interactions from Textual Descriptionscited by 60
  9. 2025Seedream 2.0: A Native Chinese-English Bilingual Image Generation Foundation Modelcited by 59
  10. 2023Readout Guidance: Learning Control from Diffusion Featurescited by 52
  11. 2024Commonsense-T2I Challenge: Can Text-to-Image Generation Models Understand Commonsense?cited by 52
  12. 2025A Fourier Space Perspective on Diffusion Modelscited by 48
38 more
  1. 2024Dynamic Prompt Optimizing for Text-to-Image Generationcited by 45
  2. 2024Revisiting Text-to-Image Evaluation with Gecko: On Metrics, Prompts, and Human Ratingscited by 44
  3. 2024Paint by Inpaint: Learning to Add Image Objects by Removing Them Firstcited by 39
  4. 2025OmniPaint: Mastering Object-Oriented Editing via Disentangled Insertion-Removal Inpaintingcited by 38
  5. 2025OmniEraser: Remove Objects and Their Effects in Images with Paired Video-Frame Datacited by 35
  6. 2024DCTdiff: Intriguing Properties of Image Generative Modeling in the DCT Spacecited by 32
  7. 2026Representation Fr'echet Loss for Visual Generationcited by 32
  8. 2024On the Challenges and Opportunities in Generative AIcited by 31
  9. 2025Diffusion Tree Sampling: Scalable inference-time alignment of diffusion modelscited by 31
  10. 2024Getting it Right: Improving Spatial Consistency in Text-to-Image Modelscited by 30
  11. 2024Erasing Concepts from Text-to-Image Diffusion Models with Few-shot Unlearningcited by 29
  12. 2024STAR: Scale-wise Text-conditioned AutoRegressive image generationcited by 28
  13. 2025A Survey of Multimodal Hallucination Evaluation and Detectioncited by 26
  14. 2025SynthID-Image: Image watermarking at internet scalecited by 26
  15. 2025SmartEraser: Remove Anything from Images using Masked-Region Guidancecited by 24
  16. 2025How Compositional Generalization and Creativity Improve as Diffusion Models are Trainedcited by 24
  17. 2025KAD: No More FAD! An Effective and Efficient Evaluation Metric for Audio Generationcited by 24
  18. 2025Flow Matching for Medical Image Synthesis: Bridging the Gap Between Speed and Qualitycited by 24
  19. 2025UniTEX: Universal High Fidelity Generative Texturing for 3D Shapescited by 23
  20. 2024Collaborative Control for Geometry-Conditioned PBR Image Generationcited by 22
  21. 2024Rethinking Training for De-biasing Text-to-Image Generation: Unlocking the Potential of Stable Diffusioncited by 22
  22. 2024Bigger is not Always Better: Scaling Properties of Latent Diffusion Modelscited by 21
  23. 2024Beyond FVD: Enhanced Evaluation Metrics for Video Generation Qualitycited by 21
  24. 2025Noise-Consistent Siamese-Diffusion for Medical Image Synthesis and Segmentationcited by 21
  25. 2024Neural Image Compression with Text-guided Encoding for both Pixel-level and Perceptual Fidelitycited by 20
  26. 2024SATO: Stable Text-to-Motion Frameworkcited by 20
  27. 2025Aligning Text-to-Music Evaluation with Human Preferencescited by 20
  28. 2025RomanTex: Decoupling 3D-aware Rotary Positional Embedded Multi-Attention Network for Texture Synthesiscited by 20
  29. 2025Population-Aligned Persona Generation for LLM-based Social Simulationcited by 20
  30. 2026AutoFigure: Generating and Refining Publication-Ready Scientific Illustrationscited by 20
  31. 2024CLIPAway: Harmonizing Focused Embeddings for Removing Objects via Diffusion Modelscited by 19
  32. 2024Fr'echet Radiomic Distance (FRD): A Versatile Metric for Comparing Medical Imaging Datasetscited by 19
  33. 2025A Survey on 3D Gaussian Splatting Applications: Segmentation, Editing, and Generationcited by 19
  34. 2024Pruning then Reweighting: Towards Data-Efficient Training of Diffusion Modelscited by 17
  35. 2024Presto! Distilling Steps and Layers for Accelerating Music Generationcited by 16
  36. 2025Skyfall-GS: Synthesizing Immersive 3D Urban Scenes from Satellite Imagerycited by 16
  37. 2024LeFusion: Controllable Pathology Synthesis via Lesion-Focused Diffusion Modelscited by 15
  38. 2024A Practical Guide to Sample-based Statistical Distances for Evaluating Generative Models in Sciencecited by 14

50 most cited of 303 All 303 (JSON)