# Evaluating the Diversity of AI-Generated Content with Diversity Profiles

arXiv: https://arxiv.org/abs/2608.17731 · HTML: https://mlsanity.com/paper/2608.17731/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 15 of its 35 references that are in the map.
- First posted on arXiv 18 Aug 2026.

## Builds on (most cited first, 15 of 15)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Qwen3 Technical Report](https://mlsanity.com/paper/2505.09388/) (2025)
- [Constructing A Multi-hop QA Dataset for Comprehensive Evaluation of Reasoning Steps](https://mlsanity.com/paper/2011.01060/) (2020)
- [The Vendi Score: A Diversity Evaluation Metric for Machine Learning](https://mlsanity.com/paper/2210.02410/) (2022)
- [Evaluating the Evaluation of Diversity in Natural Language Generation](https://mlsanity.com/paper/2004.02990/) (2020)
- [How Much Space Has Been Explored? Measuring the Chemical Space Covered by Databases and Machine-Generated Molecules](https://mlsanity.com/paper/2112.12542/) (2021)
- [Towards a Scalable Reference-Free Evaluation of Generative Models](https://mlsanity.com/paper/2407.02961/) (2024)
- [Cousins Of The Vendi Score: A Family Of Similarity-Based Diversity Metrics For Science And Machine Learning](https://mlsanity.com/paper/2310.12952/) (2023)
- [Measuring Data Diversity for Instruction Tuning: A Systematic Analysis and A Reliable Metric](https://mlsanity.com/paper/2502.17184/) (2025)
- [ChemGAN challenge for drug discovery: can AI reproduce natural chemical diversity?](https://mlsanity.com/paper/1708.08227/) (2017)
- [Metric Space Magnitude for Evaluating the Diversity of Latent Representations](https://mlsanity.com/paper/2311.16054/) (2023)
- [Position: All Current Generative Fidelity and Diversity Metrics are Flawed](https://mlsanity.com/paper/2505.22450/) (2025)
- [An Axiomatic Analysis of Diversity Evaluation Metrics: Introducing the Rank-Biased Utility Metric](https://mlsanity.com/paper/1805.02334/) (2018)
- [Challenges of Generating Structurally Diverse Graphs](https://mlsanity.com/paper/2409.18859/) (2024)
- [Measuring Diversity: Axioms and Challenges](https://mlsanity.com/paper/2410.14556/) (2024)

## Abstract (arXiv)

Diversity is a fundamental criterion for evaluating generative artificial intelligence (AI) systems, yet its measurement remains inherently ambiguous. Existing approaches typically represent generated samples in an embedding space, compute pairwise distances or similarities, and aggregate them into a single scalar score. Such scalar summaries are convenient, but they often encode different inductive biases and may yield contradictory rankings of the same sample sets. In this paper, we argue that diversity evaluation for AI-generated content is intrinsically under-specified when reduced to a single number. We first review representative diversity metrics, and then diagnose their limitations from two complementary perspectives: an axiomatic analysis showing that no representative scalar metric satisfies all desirable properties simultaneously, and an empirical analysis showing that high-dimensional representation spaces can induce concentrated, modality-dependent distance distributions. To address these issues, we propose diversity profiles: curve-valued, condition-aware summaries that evaluate a parameterized diversity family across a range of thresholds, scales, exponents, or orders under a specified representation and distance or kernel function. Diversity profiles reveal whether a comparison is robust across resolutions or instead depends on an arbitrary parameter choice. We instantiate profiles for several representative metric families and demonstrate their practical use in generative AI evaluation. Overall, diversity profiles provide a more transparent and resolution-aware framework for comparing the diversity of AI-generated content.