mlsanityData to 14 Oct 2026

Position: All Current Generative Fidelity and Diversity Metrics are Flawed

arXiv 2505.22450 ↗First posted 28 May 2025cs.LGcited by 12

Not on the map. Why?

Abstract (arXiv)

Any method's development and practical application is limited by our ability to measure its reliability. The popularity of generative modeling emphasizes the importance of good synthetic data metrics. Unfortunately, previous works have found many failure cases in current metrics, for example lack of outlier robustness and unclear lower and upper bounds. We propose a list of desiderata for synthetic data metrics, and a suite of sanity checks: carefully chosen simple experiments that aim to detect specific and known generative modeling failure modes. Based on these desiderata and the results of our checks, we arrive at our position: all current generative fidelity and diversity metrics are flawed. This significantly hinders practical use of synthetic data. Our aim is to convince the research community to spend more effort in developing metrics, instead of models. Additionally, through analyzing how current metrics fail, we provide practitioners with guidelines on how these metrics should (not) be used.

Key facts. Cited by 12 papers in the map (data through 14 Oct 2026). Builds on 27 of its 43 references that are in the map. First posted on arXiv 28 May 2025.

Builds on

27 of 43 references.

References · 27

  1. 2018A Style-Based Generator Architecture for Generative Adversarial Networkscited by 6,013
  2. 2016Improved Techniques for Training GANscited by 2,970
  3. 2019Improved Precision and Recall Metric for Assessing Generative Modelscited by 944
  4. 2020Reliable Fidelity and Diversity Metrics for Generative Modelscited by 434
  5. 2018Assessing Generative Models via Precision and Recallcited by 406
  6. 2022TabDDPM: Modelling Tabular Data with Diffusion Modelscited by 377
  7. 2023Rethinking FID: Towards a Better Evaluation Metric for Image Generationcited by 310
  8. 2022Language Models are Realistic Tabular Data Generatorscited by 294
  9. 2015A note on the evaluation of generative modelscited by 292
  10. 2023Imitating Human Behaviour with Diffusion Modelscited by 263
  11. 2018Pros and Cons of GAN Evaluation Measurescited by 224
  12. 2021How Faithful is your Synthetic Data? Sample-level Metrics for Evaluating and Auditing Generative Modelscited by 196
15 more
  1. 2022RenderDiffusion: Image Diffusion for 3D Reconstruction, Inpainting and Generationcited by 162
  2. 2021Pros and Cons of GAN Evaluation Measures: New Developmentscited by 123
  3. 2021DECAF: Generating Fair Synthetic Data Using Causally-Aware Generative Networkscited by 109
  4. 2022AIM: An Adaptive and Iterative Mechanism for Differentially Private Synthetic Datacited by 90
  5. 2018An empirical study on evaluation metrics of generative adversarial networkscited by 81
  6. 2023Generative Table Pre-training Empowers Models for Tabular Predictioncited by 55
  7. 2023Can You Rely on Your Model Evaluation? Improving Model Evaluation with Synthetic Test Datacited by 32
  8. 2021Manifold Topology Divergence: a Framework for Comparing Data Manifoldscited by 31
  9. 2023Curvature Filtrations for Graph Generative Model Evaluationcited by 19
  10. 2021Conditional Synthetic Data Generation for Robust Machine Learning Applications with Limited Pandemic Datacited by 18
  11. 2023TopP&R: Robust Support Estimation Approach for Evaluating Fidelity and Diversity in Generative Modelscited by 14
  12. 2023Probabilistic Precision and Recall Towards Reliable Evaluation of Generative Modelscited by 11
  13. 2020On the Theoretical Equivalence of Several Trade-Off Curves Assessing Statistical Proximitycited by 6
  14. 2023Emergent Asymmetry of Precision and Recall for Measuring Fidelity and Diversity of Generative Models in High Dimensionscited by 5
  15. 2024Unifying and extending Precision Recall metrics for assessing generative modelscited by 1

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 2; 2026: 10.

12 papers.

  1. 2026Mechanism Design Is Not Enough: Prosocial Agents for Cooperative AIcited by 3
  2. 2026Diffusion models recover accurate mixture weights despite score function insensitivitycited by 3
  3. 2025Enhanced Generative Model Evaluation with Clipped Density and Coveragecited by 1
  4. 2026Dataset Diversity Metrics and Impact on Classification Modelscited by 1
  5. 2026Scalable and Interpretable Representation Alignment with Ordinal Similaritycited by 1
  6. 2025seqme: a Python library for evaluating biological sequence designcited by 0
  7. 2026GICDM: Mitigating Hubness for Reliable Distance-Based Generative Model Evaluationcited by 0
  8. 2026Not All Prediction Targets Keep Training-Free Diffusion Guidance on the Manifoldcited by 0
  9. 2026Fidelity-Diversity Metrics for Textcited by 0
  10. 2026Clinical Communication Processing with Models Trained on LLM-Generated Synthetic Data: A Structured Survey and Novel Application Case Studiescited by 0
  11. 2026An Investigation of the NeurIPS and ICML 2025 Position Trackscited by 0
  12. 2026Evaluating the Diversity of AI-Generated Content with Diversity Profilescited by 0