# Measuring Diversity in Synthetic Datasets

arXiv: https://arxiv.org/abs/2502.08512 · HTML: https://mlsanity.com/paper/2502.08512/

## Key facts

- Cited by 5 papers in the map (data through 14 Oct 2026).
- Placed in LLM synthetic data generation, one of 13 directions in Synthetic data and self-improvement for LLMs.
- Builds on 60 of its 90 references that are in the map.
- First posted on arXiv 12 Feb 2025.

## Placed in

- Synthetic data and self-improvement for LLMs / [LLM synthetic data generation](https://mlsanity.com/synth/llm-synthetic-data-generation/)

## Builds on (most cited first, 30 of 60)

- [GPT-4 Technical Report](https://mlsanity.com/paper/2303.08774/) (2023)
- [RoBERTa: A Robustly Optimized BERT Pretraining Approach](https://mlsanity.com/paper/1907.11692/) (2019)
- [Llama 2: Open Foundation and Fine-Tuned Chat Models](https://mlsanity.com/paper/2307.09288/) (2023)
- [Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks](https://mlsanity.com/paper/1908.10084/) (2019)
- [Generative Adversarial Networks](https://mlsanity.com/paper/2203.00667/) (2022)
- [mixup: Beyond Empirical Risk Minimization](https://mlsanity.com/paper/1710.09412/) (2017)
- [The Curious Case of Neural Text Degeneration](https://mlsanity.com/paper/1904.09751/) (2019)
- [Self-Instruct: Aligning Language Models with Self-Generated Instructions](https://mlsanity.com/paper/2212.10560/) (2022)
- [Character-level Convolutional Networks for Text Classification](https://mlsanity.com/paper/1509.01626/) (2015)
- [SimCSE: Simple Contrastive Learning of Sentence Embeddings](https://mlsanity.com/paper/2104.08821/) (2021)
- [SpecAugment: A Simple Data Augmentation Method for Automatic Speech Recognition](https://mlsanity.com/paper/1904.08779/) (2019)
- [A Diversity-Promoting Objective Function for Neural Conversation Models](https://mlsanity.com/paper/1510.03055/) (2015)
- [Beyond Accuracy: Behavioral Testing of NLP models with CheckList](https://mlsanity.com/paper/2005.04118/) (2020)
- [Improved Precision and Recall Metric for Assessing Generative Models](https://mlsanity.com/paper/1904.06991/) (2019)
- [ChatGPT Outperforms Crowd-Workers for Text-Annotation Tasks](https://mlsanity.com/paper/2303.15056/) (2023)
- [A Prompt Pattern Catalog to Enhance Prompt Engineering with ChatGPT](https://mlsanity.com/paper/2302.11382/) (2023)
- [Texygen: A Benchmarking Platform for Text Generation Models](https://mlsanity.com/paper/1802.01886/) (2018)
- [MAUVE: Measuring the Gap Between Neural Text and Human Text using Divergence Frontiers](https://mlsanity.com/paper/2102.01454/) (2021)
- [Reliable Fidelity and Diversity Metrics for Generative Models](https://mlsanity.com/paper/2002.09797/) (2020)
- [SeqGAN: Sequence Generative Adversarial Nets with Policy Gradient](https://mlsanity.com/paper/1609.05473/) (2016)
- [The Vendi Score: A Diversity Evaluation Metric for Machine Learning](https://mlsanity.com/paper/2210.02410/) (2022)
- [Additive Margin Softmax for Face Verification](https://mlsanity.com/paper/1801.05599/) (2018)
- [On LLMs-Driven Synthetic Data Generation, Curation, and Evaluation: A Survey](https://mlsanity.com/paper/2406.15126/) (2024)
- [Large Language Model as Attributed Training Data Generator: A Tale of Diversity and Bias](https://mlsanity.com/paper/2306.15895/) (2023)
- [ZeroGen: Efficient Zero-shot Learning via Dataset Generation](https://mlsanity.com/paper/2202.07922/) (2022)
- [Is GPT-3 a Good Data Annotator?](https://mlsanity.com/paper/2212.10450/) (2022)
- [Does Writing with Language Models Reduce Content Diversity?](https://mlsanity.com/paper/2309.05196/) (2023)
- [Advancing LLM Reasoning Generalists with Preference Trees](https://mlsanity.com/paper/2404.02078/) (2024)
- [GPT3Mix: Leveraging Large-scale Language Models for Text Augmentation](https://mlsanity.com/paper/2104.08826/) (2021)
- [Online normalizer calculation for softmax](https://mlsanity.com/paper/1805.02867/) (2018)

## Built on by (most cited first, 5 of 5)

- [A Survey on Evaluating Quality and Trustworthiness in LLM-Generated Data](https://mlsanity.com/paper/2601.17717/) (2026)
- [SynQuE: Estimating Synthetic Dataset Quality Without Annotations](https://mlsanity.com/paper/2511.03928/) (2025)
- [Exploring More to Solve More: Boosting Diversity in Text Diffusion Models via Entropy-Based Guidance](https://mlsanity.com/paper/2608.00024/) (2026)
- [HAMMER: Hamiltonian Curiosity Augmented Large Language Model Reinforcement](https://mlsanity.com/paper/2509.25240/) (2025)
- [Measuring Dataset Diversity from a Geometric Perspective](https://mlsanity.com/paper/2602.09340/) (2026)

## Abstract (arXiv)

Large language models (LLMs) are widely adopted to generate synthetic datasets for various natural language processing (NLP) tasks, such as text classification and summarization. However, accurately measuring the diversity of these synthetic datasets-an aspect crucial for robust model performance-remains a significant challenge. In this paper, we introduce DCScore, a novel method for measuring synthetic dataset diversity from a classification perspective. Specifically, DCScore formulates diversity evaluation as a sample classification task, leveraging mutual relationships among samples. We further provide theoretical verification of the diversity-related axioms satisfied by DCScore, highlighting its role as a principled diversity evaluation method. Experimental results on synthetic datasets reveal that DCScore enjoys a stronger correlation with multiple diversity pseudo-truths of evaluated datasets, underscoring its effectiveness. Moreover, both empirical and theoretical evidence demonstrate that DCScore substantially reduces computational costs compared to existing methods. Code is available at: https://github.com/bluewhalelab/dcscore.