mlsanityData to 14 Oct 2026

What FID Hides: Detecting, Ranking, and Diagnosing Deviations in Generative Evaluation

arXiv 2608.24881 ↗First posted 25 Aug 2026stat.ML cs.LGcited by 0

Not on the map. Why?

Abstract (arXiv)

Generative models are commonly ranked by Fr'echet Inception Distance (FID) and Kernel Inception Distance (KID), yet FID's first-two-moment summary can miss distributional differences, and a reported scalar gap alone is not a calibrated test against sampling variation. FID's moment restriction has concrete consequences: on ImageNet, visually unrecognizable images optimized only to match the reference Inception mean and covariance obtain FID 24.7 versus 58.6 for held-out real images (lower is better). Moreover, FID and KID are scalar discrepancies that are unchanged when the two samples are exchanged and therefore do not encode the direction of a dispersion change: under-dispersion, as can occur in mode collapse, versus over-dispersion. We introduce ZID (Z-resolved Integrated Diagnostic), which combines six standardized location- and dispersion-sensitive arms from a rank graph (RISE) and Gaussian kernels (GPK at two bandwidths). Rather than asking one scalar to serve incompatible roles, ZID reports three linked outputs: an index for ranking departure magnitude, a permutation p-value for testing distributional equality, and a signed dispersion readout for diagnosis. In controlled experiments, ZID detects a broad range of departures, and its score tracks increasing severity along the corresponding sweeps, including cases in which FID is flat or reversed. On DiT-XL/2 and SiT-XL/2 guidance sweeps, ZID detects departure from real data, and its signed readout labels the high-guidance diversity collapse as under-dispersion.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 30 of its 38 references that are in the map. First posted on arXiv 25 Aug 2026.

Builds on

30 of 38 references.

References · 30

  1. 2020Denoising Diffusion Probabilistic Modelscited by 21,076
  2. 2020Denoising Diffusion Implicit Modelscited by 9,781
  3. 2014ImageNet Large Scale Visual Recognition Challengecited by 8,498
  4. 2021Emerging Properties in Self-Supervised Vision Transformerscited by 6,968
  5. 2022Scalable Diffusion Models with Transformerscited by 6,054
  6. 2018A Style-Based Generator Architecture for Generative Adversarial Networkscited by 6,013
  7. 2022Classifier-Free Diffusion Guidancecited by 5,847
  8. 2015Rethinking the Inception Architecture for Computer Visioncited by 5,475
  9. 2019Analyzing and Improving the Image Quality of StyleGANcited by 3,588
  10. 2018Large Scale GAN Training for High Fidelity Natural Image Synthesiscited by 2,626
  11. 2018Demystifying MMD GANscited by 1,293
  12. 2020Training Generative Adversarial Networks with Limited Datacited by 1,186
18 more
  1. 2019Improved Precision and Recall Metric for Assessing Generative Modelscited by 944
  2. 2024SiT: Exploring Flow and Diffusion-based Generative Models with Scalable Interpolant Transformerscited by 737
  3. 2020Reliable Fidelity and Diversity Metrics for Generative Modelscited by 434
  4. 2018Assessing Generative Models via Precision and Recallcited by 406
  5. 2022The Vendi Score: A Diversity Evaluation Metric for Machine Learningcited by 311
  6. 2023Rethinking FID: Towards a Better Evaluation Metric for Image Generationcited by 310
  7. 2016Revisiting Classifier Two-Sample Testscited by 268
  8. 2022The Role of ImageNet Classes in Fr'echet Inception Distancecited by 213
  9. 2019Effectively Unbiased FID and Inception Score and where to find themcited by 165
  10. 2013A new graph-based two-sample test for multivariate and object datacited by 24
  11. 2018Cauchy combination test: a powerful test with analytic p-value calculation under arbitrary dependency structurescited by 20
  12. 2020Evaluation Metrics for Conditional Image Generationcited by 19
  13. 2021Evaluating generative networks using Gaussian mixtures of image featurescited by 11
  14. 2022On the Robustness of Quality Measures for GANscited by 11
  15. 2026MIND: Monge Inception Distance for Generative Models Evaluationcited by 10
  16. 2021Compound Frechet Inception Distance for Quality Assessment of GAN Created Imagescited by 8
  17. 2020Generalized Kernel Two-Sample Testscited by 5
  18. 2025Assessing the Distributional Fidelity of Synthetic Chest X-rays using the Embedded Characteristic Scorecited by 1

Built on by

Not cited yet.