# Position: All Current Generative Fidelity and Diversity Metrics are Flawed arXiv: https://arxiv.org/abs/2505.22450 ยท HTML: https://mlsanity.com/paper/2505.22450/ ## Key facts - Cited by 12 papers in the map (data through 14 Oct 2026). - Builds on 27 of its 43 references that are in the map. - First posted on arXiv 28 May 2025. ## Builds on (most cited first, 27 of 27) - [A Style-Based Generator Architecture for Generative Adversarial Networks](https://mlsanity.com/paper/1812.04948/) (2018) - [Improved Techniques for Training GANs](https://mlsanity.com/paper/1606.03498/) (2016) - [Improved Precision and Recall Metric for Assessing Generative Models](https://mlsanity.com/paper/1904.06991/) (2019) - [Reliable Fidelity and Diversity Metrics for Generative Models](https://mlsanity.com/paper/2002.09797/) (2020) - [Assessing Generative Models via Precision and Recall](https://mlsanity.com/paper/1806.00035/) (2018) - [TabDDPM: Modelling Tabular Data with Diffusion Models](https://mlsanity.com/paper/2209.15421/) (2022) - [Rethinking FID: Towards a Better Evaluation Metric for Image Generation](https://mlsanity.com/paper/2401.09603/) (2023) - [Language Models are Realistic Tabular Data Generators](https://mlsanity.com/paper/2210.06280/) (2022) - [A note on the evaluation of generative models](https://mlsanity.com/paper/1511.01844/) (2015) - [Imitating Human Behaviour with Diffusion Models](https://mlsanity.com/paper/2301.10677/) (2023) - [Pros and Cons of GAN Evaluation Measures](https://mlsanity.com/paper/1802.03446/) (2018) - [How Faithful is your Synthetic Data? Sample-level Metrics for Evaluating and Auditing Generative Models](https://mlsanity.com/paper/2102.08921/) (2021) - [RenderDiffusion: Image Diffusion for 3D Reconstruction, Inpainting and Generation](https://mlsanity.com/paper/2211.09869/) (2022) - [Pros and Cons of GAN Evaluation Measures: New Developments](https://mlsanity.com/paper/2103.09396/) (2021) - [DECAF: Generating Fair Synthetic Data Using Causally-Aware Generative Networks](https://mlsanity.com/paper/2110.12884/) (2021) - [AIM: An Adaptive and Iterative Mechanism for Differentially Private Synthetic Data](https://mlsanity.com/paper/2201.12677/) (2022) - [An empirical study on evaluation metrics of generative adversarial networks](https://mlsanity.com/paper/1806.07755/) (2018) - [Generative Table Pre-training Empowers Models for Tabular Prediction](https://mlsanity.com/paper/2305.09696/) (2023) - [Can You Rely on Your Model Evaluation? Improving Model Evaluation with Synthetic Test Data](https://mlsanity.com/paper/2310.16524/) (2023) - [Manifold Topology Divergence: a Framework for Comparing Data Manifolds](https://mlsanity.com/paper/2106.04024/) (2021) - [Curvature Filtrations for Graph Generative Model Evaluation](https://mlsanity.com/paper/2301.12906/) (2023) - [Conditional Synthetic Data Generation for Robust Machine Learning Applications with Limited Pandemic Data](https://mlsanity.com/paper/2109.06486/) (2021) - [TopP&R: Robust Support Estimation Approach for Evaluating Fidelity and Diversity in Generative Models](https://mlsanity.com/paper/2306.08013/) (2023) - [Probabilistic Precision and Recall Towards Reliable Evaluation of Generative Models](https://mlsanity.com/paper/2309.01590/) (2023) - [On the Theoretical Equivalence of Several Trade-Off Curves Assessing Statistical Proximity](https://mlsanity.com/paper/2006.11809/) (2020) - [Emergent Asymmetry of Precision and Recall for Measuring Fidelity and Diversity of Generative Models in High Dimensions](https://mlsanity.com/paper/2306.09618/) (2023) - [Unifying and extending Precision Recall metrics for assessing generative models](https://mlsanity.com/paper/2405.01611/) (2024) ## Built on by (most cited first, 12 of 12) - [Mechanism Design Is Not Enough: Prosocial Agents for Cooperative AI](https://mlsanity.com/paper/2605.08426/) (2026) - [Diffusion models recover accurate mixture weights despite score function insensitivity](https://mlsanity.com/paper/2607.15485/) (2026) - [Enhanced Generative Model Evaluation with Clipped Density and Coverage](https://mlsanity.com/paper/2507.01761/) (2025) - [Dataset Diversity Metrics and Impact on Classification Models](https://mlsanity.com/paper/2603.15276/) (2026) - [Scalable and Interpretable Representation Alignment with Ordinal Similarity](https://mlsanity.com/paper/2606.16379/) (2026) - [seqme: a Python library for evaluating biological sequence design](https://mlsanity.com/paper/2511.04239/) (2025) - [GICDM: Mitigating Hubness for Reliable Distance-Based Generative Model Evaluation](https://mlsanity.com/paper/2602.16449/) (2026) - [Not All Prediction Targets Keep Training-Free Diffusion Guidance on the Manifold](https://mlsanity.com/paper/2607.00647/) (2026) - [Fidelity-Diversity Metrics for Text](https://mlsanity.com/paper/2607.04563/) (2026) - [Clinical Communication Processing with Models Trained on LLM-Generated Synthetic Data: A Structured Survey and Novel Application Case Studies](https://mlsanity.com/paper/2608.05993/) (2026) - [An Investigation of the NeurIPS and ICML 2025 Position Tracks](https://mlsanity.com/paper/2608.16894/) (2026) - [Evaluating the Diversity of AI-Generated Content with Diversity Profiles](https://mlsanity.com/paper/2608.17731/) (2026) ## Abstract (arXiv) Any method's development and practical application is limited by our ability to measure its reliability. The popularity of generative modeling emphasizes the importance of good synthetic data metrics. Unfortunately, previous works have found many failure cases in current metrics, for example lack of outlier robustness and unclear lower and upper bounds. We propose a list of desiderata for synthetic data metrics, and a suite of sanity checks: carefully chosen simple experiments that aim to detect specific and known generative modeling failure modes. Based on these desiderata and the results of our checks, we arrive at our position: all current generative fidelity and diversity metrics are flawed. This significantly hinders practical use of synthetic data. Our aim is to convince the research community to spend more effort in developing metrics, instead of models. Additionally, through analyzing how current metrics fail, we provide practitioners with guidelines on how these metrics should (not) be used.