mlsanityData to 7 Oct 2026

The Vendi Score: A Diversity Evaluation Metric for Machine Learning

arXiv 2210.02410 ↗First posted 5 Oct 2022cs.LG cond-mat.mtrl-sci stat.MLcited by 299

Not on the map. Why?

Cited by 56 in Diffusion models, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

Diversity is an important criterion for many areas of machine learning (ML), including generative modeling and dataset curation. However, existing metrics for measuring diversity are often domain-specific and limited in flexibility. In this paper, we address the diversity evaluation problem by proposing the Vendi Score, which connects and extends ideas from ecology and quantum statistical mechanics to ML. The Vendi Score is defined as the exponential of the Shannon entropy of the eigenvalues of a similarity matrix. This matrix is induced by a user-defined similarity function applied to the sample to be evaluated for diversity. In taking a similarity function as input, the Vendi Score enables its user to specify any desired form of diversity. Importantly, unlike many existing metrics in ML, the Vendi Score does not require a reference dataset or distribution over samples or labels, it is therefore general and applicable to any generative model, decoding algorithm, and dataset from any domain where similarity can be defined. We showcase the Vendi Score on molecular generative modeling where we found it addresses shortcomings of the current diversity metric of choice in that domain. We also applied the Vendi Score to generative models of images and decoding algorithms of text where we found it confirms known results about diversity in those domains. Furthermore, we used the Vendi Score to measure mode collapse, a known shortcoming of generative adversarial networks (GANs). In particular, the Vendi Score revealed that even GANs that capture all the modes of a labeled dataset can be less diverse than the original dataset. Finally, the interpretability of the Vendi Score allowed us to diagnose several benchmark ML datasets for diversity, opening the door for diversity-informed data augmentation.

Key facts. Cited by 299 papers in the map (data through 7 Oct 2026). Builds on 43 of its 68 references that are in the map. First posted on arXiv 5 Oct 2022.

Builds on

43 of 68 references.

References · 43

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  2. 2014Microsoft COCO: Common Objects in Contextcited by 17,498
  3. 2020Denoising Diffusion Implicit Modelscited by 9,655
  4. 2014ImageNet Large Scale Visual Recognition Challengecited by 8,441
  5. 2021Diffusion Models Beat GANs on Image Synthesiscited by 8,185
  6. 2018A Style-Based Generator Architecture for Generative Adversarial Networkscited by 5,986
  7. 2015Rethinking the Inception Architecture for Computer Visioncited by 5,451
  8. 2017Fashion-MNIST: a Novel Image Dataset for Benchmarking Machine Learning Algorithmscited by 4,283
  9. 2021Improved Denoising Diffusion Probabilistic Modelscited by 3,736
  10. 2014Deep Learning Face Attributes in the Wildcited by 3,679
  11. 2019Analyzing and Improving the Image Quality of StyleGANcited by 3,580
  12. 2016Improved Techniques for Training GANscited by 2,938
31 more
  1. 2017A Broad-Coverage Challenge Corpus for Sentence Understanding through Inferencecited by 2,665
  2. 2019HuggingFace's Transformers: State-of-the-art Natural Language Processingcited by 2,201
  3. 2021SimCSE: Simple Contrastive Learning of Sentence Embeddingscited by 2,168
  4. 2015Unsupervised Representation Learning with Deep Convolutional Generative Adversarial Networkscited by 2,048
  5. 2015LSUN: Construction of a Large-scale Image Dataset using Deep Learning with Humans in the Loopcited by 1,350
  6. 2015A Diversity-Promoting Objective Function for Neural Conversation Modelscited by 1,068
  7. 2019Improved Precision and Recall Metric for Assessing Generative Modelscited by 925
  8. 2022Autoregressive Image Generation using Residual Quantizationcited by 706
  9. 2020Reliable Fidelity and Diversity Metrics for Generative Modelscited by 425
  10. 2019Implicit Regularization in Deep Matrix Factorizationcited by 402
  11. 2018Assessing Generative Models via Precision and Recallcited by 401
  12. 2018Molecular Sets (MOSES): A Benchmarking Platform for Molecular Generation Modelscited by 270
  13. 2020Very Deep VAEs Generalize Autoregressive Models and Can Outperform Them on Imagescited by 261
  14. 2020Unsupervised Quality Estimation for Neural Machine Translationcited by 210
  15. 2016Unrolled Generative Adversarial Networkscited by 208
  16. 2017VEEGAN: Reducing Mode Collapse in GANs using Implicit Variational Learningcited by 199
  17. 2018Fr'echet ChemNet Distance: A metric for generative models for molecules in drug discoverycited by 191
  18. 2020The Multilingual Amazon Reviews Corpuscited by 142
  19. 2019Mixture Models for Diverse Machine Translation: Tricks of the Tradecited by 77
  20. 2020Diverse Image Generation via Self-Conditioned GANscited by 54
  21. 2021Densely connected normalizing flowscited by 49
  22. 2022Information Theory with Kernel Methodscited by 37
  23. 2019Machine Learning for Scent: Learning Generalizable Perceptual Representations of Small Moleculescited by 28
  24. 2020Diversity and Inclusion Metrics in Subset Selectioncited by 27
  25. 2019Prescribed Generative Adversarial Networkscited by 23
  26. 2017ChemGAN challenge for drug discovery: can AI reproduce natural chemical diversity?cited by 21
  27. 2022Multiple Importance Sampling ELBO and Deep Ensembles of Variational Approximationscited by 12
  28. 2018GILBO: One Metric to Measure Them Allcited by 9
  29. 2019Revisiting Precision and Recall Definition for Generative Model Evaluationcited by 9
  30. 2018Eval all, trust a few, do wrong to none: Comparing sentence generation modelscited by 8
  31. 2019GAIT: A Geometric Approach to Information Theorycited by 5

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 1; 2023: 16; 2024: 59; 2025: 78; 2026: 145.

299 papers.

Sort:

By area:

  1. 2023Exposing flaws of generative model evaluation metrics and their unfair treatment of diffusion modelscited by 199
  2. 2025T2I-R1: Reinforcing Image Generation with Collaborative Semantic-level and Token-level CoTcited by 162
  3. 2024Kernel Language Entropy: Fine-grained Uncertainty Quantification for LLMs from Semantic Similaritiescited by 156
  4. 2023CADS: Unleashing the Diversity of Diffusion Models through Condition-Annealed Samplingcited by 124
  5. 2023Navigating Text-To-Image Customization: From LyCORIS Fine-Tuning to Model Evaluationcited by 108
  6. 2024A Survey of Multimodal Large Language Model from A Data-centric Perspectivecited by 73
  7. 2023Self-Evolved Diverse Data Sampling for Efficient Instruction Tuningcited by 52
  8. 2025FineVision: Open Data Is All You Needcited by 51
  9. 2025Prismatic Synthesis: Gradient-based Data Diversification Boosts Generalization in LLM Reasoningcited by 38
  10. 2023Multilevel Diffusion: Infinite Dimensional Score-Based Diffusion Models for Image Generationcited by 37
  11. 2024Consistency-diversity-realism Pareto fronts of conditional image generative modelscited by 35
  12. 2024Motion Planning Diffusion: Learning and Adapting Robot Motion Planning with Diffusion Modelscited by 32
38 more
  1. 2024Exploring Autonomous Agents through the Lens of Large Language Models: A Reviewcited by 30
  2. 2024Beyond Aesthetics: Cultural Competence in Text-to-Image Modelscited by 30
  3. 2024Unleashing the Power of Data Tsunami: A Comprehensive Survey on Data Assessment and Selection for Instruction Tuning of Language Modelscited by 29
  4. 2024Diversity Measurement and Subset Selection for Instruction Tuning Datasetscited by 28
  5. 2024MiniPLM: Knowledge Distillation for Pre-Training Language Modelscited by 27
  6. 2023Diffusion Illusions: Hiding Images in Plain Sightcited by 26
  7. 2024Benchmarking Large Language Models on Controllable Generation under Diversified Instructionscited by 26
  8. 2024Position: Measure Dataset Diversity, Don't Just Claim Itcited by 26
  9. 2024Towards a Scalable Reference-Free Evaluation of Generative Modelscited by 25
  10. 2024DOLLAR: Few-Step Video Generation via Distillation and Latent Reward Optimizationcited by 25
  11. 2023Cousins Of The Vendi Score: A Family Of Similarity-Based Diversity Metrics For Science And Machine Learningcited by 24
  12. 2024Improving Geo-diversity of Generated Images with Contextualized Vendi Score Guidancecited by 24
  13. 2024Image Generation Diversity Issues and How to Tame Themcited by 22
  14. 2024In-Context Example Selection via Similarity Search Improves Low-Resource Machine Translationcited by 21
  15. 2024Shielded Diffusion: Generating Novel and Diverse Images using Sparse Repellencycited by 21
  16. 2024Automating the Search for Artificial Life with Foundation Modelscited by 20
  17. 2025LeMat-GenBench: A Unified Evaluation Framework for Crystal Generative Modelscited by 20
  18. 2023Beyond Scale: The Diversity Coefficient as a Data Quality Metric for Variability in Natural Language Datacited by 19
  19. 2024DiffuSolve: Diffusion-based Solver for Non-convex Trajectory Optimizationcited by 19
  20. 2022Measuring Datacited by 18
  21. 2024An Interpretable Evaluation of Entropy-based Novelty of Generative Modelscited by 17
  22. 2023Metric Space Magnitude for Evaluating the Diversity of Latent Representationscited by 16
  23. 2024Unveiling Differences in Generative Models: A Scalable Differential Clustering Approachcited by 16
  24. 2024Towards Geographic Inclusion in the Evaluation of Text-to-Image Modelscited by 16
  25. 2024Text2Chart31: Instruction Tuning for Chart Generation with Automatic Feedbackcited by 16
  26. 2026RetroAgent: From Solving to Evolving via Retrospective Dual Intrinsic Feedbackcited by 16
  27. 2024Quality-Weighted Vendi Scores And Their Application To Diverse Experimental Designcited by 15
  28. 2024ScalingFilter: Assessing Data Quality through Inverse Utilization of Scaling Lawscited by 15
  29. 2024Scendi Score: Prompt-Aware Diversity Evaluation via Schur Complement of CLIP Embeddingscited by 15
  30. 2025Flow Density Control: Generative Optimization Beyond Entropy-Regularized Fine-Tuningcited by 15
  31. 2025Unveiling Attractor Cycles in Large Language Models: A Dynamical Systems View of Successive Paraphrasingcited by 14
  32. 2025CulturalFrames: Assessing Cultural Expectation Alignment in Text-to-Image Models and Evaluation Metricscited by 14
  33. 2025SPARKE: Scalable Prompt-Aware Diversity and Novelty Guidance in Diffusion Models via RKE Scorecited by 14
  34. 2026DeepPresenter: Environment-Grounded Reflection for Agentic Presentation Generationcited by 14
  35. 2024Kaleido Diffusion: Improving Conditional Diffusion Models with Autoregressive Latent Modelingcited by 13
  36. 2024A Multi-Armed Bandit Approach to Online Selection and Evaluation of Generative Modelscited by 13
  37. 2024Be More Diverse than the Most Diverse: Optimal Mixtures of Generative Models via Mixture-UCB Bandit Algorithmscited by 13
  38. 2025Vendi-RAG: Adaptively Trading-Off Diversity And Quality Significantly Improves Retrieval Augmented Generation With LLMscited by 13

50 most cited of 299 All 299 (JSON)