mlsanityData to 14 Oct 2026

How Faithful is your Synthetic Data? Sample-level Metrics for Evaluating and Auditing Generative Models

arXiv 2102.08921 ↗First posted 17 Feb 2021cs.LG stat.MLcited by 196

Not on the map. Why?

Cited by 35 in Diffusion models, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

Devising domain- and model-agnostic evaluation metrics for generative models is an important and as yet unresolved problem. Most existing metrics, which were tailored solely to the image synthesis setup, exhibit a limited capacity for diagnosing the different modes of failure of generative models across broader application domains. In this paper, we introduce a 3-dimensional evaluation metric, (α-Precision, β-Recall, Authenticity), that characterizes the fidelity, diversity and generalization performance of any generative model in a domain-agnostic fashion. Our metric unifies statistical divergence measures with precision-recall analysis, enabling sample- and distribution-level diagnoses of model fidelity and diversity. We introduce generalization as an additional, independent dimension (to the fidelity-diversity trade-off) that quantifies the extent to which a model copies training data -- a crucial performance indicator when modeling sensitive data with requirements on privacy. The three metric components correspond to (interpretable) probabilistic quantities, and are estimated via sample-level binary classification. The sample-level nature of our metric inspires a novel use case which we call model auditing, wherein we judge the quality of individual samples generated by a (black-box) model, discarding low-quality samples and hence improving the overall model performance in a post-hoc manner.

Key facts. Cited by 196 papers in the map (data through 14 Oct 2026). Builds on 25 of its 51 references that are in the map. First posted on arXiv 17 Feb 2021.

Builds on

25 of 51 references.

References · 25

  1. 2020Denoising Diffusion Probabilistic Modelscited by 21,076
  2. 2016Improved Techniques for Training GANscited by 2,970
  3. 2018Large Scale GAN Training for High Fidelity Natural Image Synthesiscited by 2,626
  4. 2017Improved Training of Wasserstein GANscited by 2,235
  5. 2020Training Generative Adversarial Networks with Limited Datacited by 1,186
  6. 2017High-Resolution Image Synthesis and Semantic Manipulation with Conditional GANscited by 1,179
  7. 2019Improved Precision and Recall Metric for Assessing Generative Modelscited by 944
  8. 2017Wasserstein GANcited by 639
  9. 2015Unsupervised Learning of Video Representations using LSTMscited by 435
  10. 2020Reliable Fidelity and Diversity Metrics for Generative Modelscited by 434
  11. 2018Assessing Generative Models via Precision and Recallcited by 406
  12. 2015A note on the evaluation of generative modelscited by 292
13 more
  1. 2017Are GANs Created Equal? A Large-Scale Studycited by 267
  2. 2018Differentially Private Generative Adversarial Networkcited by 235
  3. 2018A Note on the Inception Scorecited by 226
  4. 2018Pros and Cons of GAN Evaluation Measurescited by 224
  5. 2017GANs Trained by a Two Time-Scale Update Rule Converge to a Local Nash Equilibriumcited by 213
  6. 2016Generative Models and Model Criticism via Optimized Maximum Mean Discrepancycited by 100
  7. 2020A Non-Parametric Test to Detect Data-Copying in Generative Modelscited by 52
  8. 2016On the Quantitative Analysis of Decoder-Based Generative Modelscited by 40
  9. 2018Eval all, trust a few, do wrong to none: Comparing sentence generation modelscited by 8
  10. 2020Hide-and-Seek Privacy Challengecited by 8
  11. 2019Investigating Under and Overfitting in Wasserstein Generative Adversarial Networkscited by 7
  12. 2020A new method for parameter estimation in probabilistic models: Minimum probability flowcited by 5
  13. 2018Quantitatively Evaluating GANs With Divergences Proposed for Trainingcited by 4

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2021: 3; 2022: 18; 2023: 40; 2024: 40; 2025: 42; 2026: 53.

196 papers.

Sort:

By area:

  1. 2022Diffusion Art or Digital Forgery? Investigating Data Replication in Diffusion Modelscited by 424
  2. 2022Convergence of denoising diffusion models under the manifold hypothesiscited by 242
  3. 2023Exposing flaws of generative model evaluation metrics and their unfair treatment of diffusion modelscited by 205
  4. 2023Mixed-Type Tabular Data Synthesis with Score-based Diffusion in Latent Spacecited by 186
  5. 2021Pros and Cons of GAN Evaluation Measures: New Developmentscited by 123
  6. 2021DECAF: Generating Fair Synthetic Data Using Causally-Aware Generative Networkscited by 109
  7. 2024Why Tabular Foundation Models Should Be a Research Prioritycited by 106
  8. 2023Toward Verifiable and Reproducible Human Evaluation for Text-to-Image Generationcited by 68
  9. 2022Synthetic Data -- what, why and how?cited by 67
  10. 2023Membership Inference Attacks against Synthetic Data through Overfitting Detectioncited by 66
  11. 2024TabDiff: a Mixed-type Diffusion Model for Tabular Data Generationcited by 63
  12. 2023On the Constrained Time-Series Generation Problemcited by 54
38 more
  1. 2023Generating Faithful Synthetic Data with Large Language Models: A Case Study in Computational Social Sciencecited by 37
  2. 2023Diffusion-based Conditional ECG Generation with Structured State Space Modelscited by 36
  3. 2023Can You Rely on Your Model Evaluation? Improving Model Evaluation with Synthetic Test Datacited by 32
  4. 2023Beyond Privacy: Navigating the Opportunities and Challenges of Synthetic Datacited by 29
  5. 2023Synthetic data, real errors: how (not) to publish and use synthetic datacited by 29
  6. 2023DiffInfinite: Large Mask-Image Synthesis via Parallel Random Patch Diffusion in Histopathologycited by 29
  7. 2024Exploring Precision and Recall to assess the quality and diversity of LLMscited by 29
  8. 2024ClavaDDPM: Multi-relational Data Synthesis with Cluster-guided Diffusion Modelscited by 28
  9. 2023TSGBench: Time Series Generation Benchmarkcited by 26
  10. 2024Towards a Scalable Reference-Free Evaluation of Generative Modelscited by 26
  11. 2023Feature Likelihood Divergence: Evaluating the Generalization of Generative Models Using Samplescited by 25
  12. 2025Sparks of Science: Hypothesis Generation Using Structured Paper Datacited by 25
  13. 2023Reimagining Synthetic Tabular Data Generation through Data-Centric AI: A Comprehensive Benchmarkcited by 23
  14. 2022Data-IQ: Characterizing subgroups with heterogeneous outcomes in tabular datacited by 22
  15. 2023Knowledge-Infused Prompting: Assessing and Advancing Clinical Text Data Generation with Large Language Modelscited by 20
  16. 2023Privacy Distillation: Reducing Re-identification Risk of Multimodal Diffusion Modelscited by 18
  17. 2024Systematic Assessment of Tabular Data Synthesiscited by 18
  18. 2025A Closer Look at Model Collapse: From a Generalization-to-Memorization Perspectivecited by 18
  19. 2023SurvivalGAN: Generating Time-to-Event Data for Survival Analysiscited by 17
  20. 2024Can I trust my fake data -- A comprehensive quality assessment framework for synthetic tabular data in healthcarecited by 17
  21. 2024An Interpretable Evaluation of Entropy-based Novelty of Generative Modelscited by 17
  22. 2023A Framework for Demonstrating Practical Quantum Advantage: Racing Quantum against Classical Generative Modelscited by 16
  23. 2024Unveiling Differences in Generative Models: A Scalable Differential Clustering Approachcited by 16
  24. 2023Aligning Synthetic Medical Images with Clinical Knowledge using Human Feedbackcited by 15
  25. 2025A Survey on Deep Learning Approaches for Tabular Data Generation: Utility, Alignment, Fidelity, Privacy, Diversity, and Beyondcited by 15
  26. 2023Challenges and Remedies to Privacy and Security in AIGC: Exploring the Potential of Privacy Computing, Blockchain, and Beyondcited by 14
  27. 2024CTSyn: A Foundation Model for Cross Tabular Data Generationcited by 14
  28. 2025TabularARGN: A Flexible and Efficient Auto-Regressive Framework for Generating High-Fidelity Synthetic Datacited by 14
  29. 2023Auditing and Generating Synthetic Data with Controllable Trust Trade-offscited by 13
  30. 2023TransFusion: Generating Long, High Fidelity Time Series using Diffusion Models with Transformerscited by 13
  31. 2024A Multi-Armed Bandit Approach to Online Selection and Evaluation of Generative Modelscited by 13
  32. 2025Riemannian Flow Matching for Brain Connectivity Matrices via Pullback Geometrycited by 13
  33. 2024Data Plagiarism Index: Characterizing the Privacy Risk of Data-Copying in Tabular Generative Modelscited by 12
  34. 2025Metrics that matter: Evaluating image quality metrics for medical image generationcited by 12
  35. 2025Position: All Current Generative Fidelity and Diversity Metrics are Flawedcited by 12
  36. 2025All that structure matches does not glittercited by 12
  37. 2023Probabilistic Precision and Recall Towards Reliable Evaluation of Generative Modelscited by 11
  38. 2023Consistent Distributed Ranking of Generative Models via Kernel Distancescited by 10

50 most cited of 196 All 196 (JSON)