mlsanityData to 7 Oct 2026

Measuring Diversity in Synthetic Datasets

arXiv 2502.08512 ↗First posted 12 Feb 2025cs.CL cs.AIcited by 5

On the map: Synthetic data and self-improvement for LLMs → LLM synthetic data generation

Abstract (arXiv)

Large language models (LLMs) are widely adopted to generate synthetic datasets for various natural language processing (NLP) tasks, such as text classification and summarization. However, accurately measuring the diversity of these synthetic datasets-an aspect crucial for robust model performance-remains a significant challenge. In this paper, we introduce DCScore, a novel method for measuring synthetic dataset diversity from a classification perspective. Specifically, DCScore formulates diversity evaluation as a sample classification task, leveraging mutual relationships among samples. We further provide theoretical verification of the diversity-related axioms satisfied by DCScore, highlighting its role as a principled diversity evaluation method. Experimental results on synthetic datasets reveal that DCScore enjoys a stronger correlation with multiple diversity pseudo-truths of evaluated datasets, underscoring its effectiveness. Moreover, both empirical and theoretical evidence demonstrate that DCScore substantially reduces computational costs compared to existing methods. Code is available at: https://github.com/bluewhalelab/dcscore.

Key facts. Cited by 5 papers in the map (data through 7 Oct 2026). Placed in LLM synthetic data generation, one of 13 directions in Synthetic data and self-improvement for LLMs. Builds on 60 of its 90 references that are in the map. First posted on arXiv 12 Feb 2025.

Builds on

60 of 90 references.

Closest ancestors · 2

Same direction, newest first

  1. 2024On LLMs-Driven Synthetic Data Generation, Curation, and Evaluation: A Survey14 Jun
  2. 2023Synthetic Data Generation with Large Language Models for Text Classification: Potential and Limitations11 Oct

Other references · 58

  1. 2023GPT-4 Technical Reportcited by 18,132
  2. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,266
  3. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  4. 2019Sentence-BERT: Sentence Embeddings using Siamese BERT-Networkscited by 9,049
  5. 2022Generative Adversarial Networkscited by 5,603
  6. 2017mixup: Beyond Empirical Risk Minimizationcited by 5,155
  7. 2019The Curious Case of Neural Text Degenerationcited by 2,881
  8. 2022Self-Instruct: Aligning Language Models with Self-Generated Instructionscited by 2,674
  9. 2015Character-level Convolutional Networks for Text Classificationcited by 2,293
  10. 2021SimCSE: Simple Contrastive Learning of Sentence Embeddingscited by 2,168
  11. 2019SpecAugment: A Simple Data Augmentation Method for Automatic Speech Recognitioncited by 1,355
  12. 2015A Diversity-Promoting Objective Function for Neural Conversation Modelscited by 1,068
38 more
  1. 2020Beyond Accuracy: Behavioral Testing of NLP models with CheckListcited by 979
  2. 2019Improved Precision and Recall Metric for Assessing Generative Modelscited by 925
  3. 2023ChatGPT Outperforms Crowd-Workers for Text-Annotation Taskscited by 849
  4. 2023A Prompt Pattern Catalog to Enhance Prompt Engineering with ChatGPTcited by 612
  5. 2018Texygen: A Benchmarking Platform for Text Generation Modelscited by 484
  6. 2021MAUVE: Measuring the Gap Between Neural Text and Human Text using Divergence Frontierscited by 466
  7. 2020Reliable Fidelity and Diversity Metrics for Generative Modelscited by 425
  8. 2016SeqGAN: Sequence Generative Adversarial Nets with Policy Gradientcited by 375
  9. 2022The Vendi Score: A Diversity Evaluation Metric for Machine Learningcited by 299
  10. 2018Additive Margin Softmax for Face Verificationcited by 275
  11. 2023Large Language Model as Attributed Training Data Generator: A Tale of Diversity and Biascited by 230
  12. 2022ZeroGen: Efficient Zero-shot Learning via Dataset Generationcited by 209
  13. 2022Is GPT-3 a Good Data Annotator?cited by 208
  14. 2024Advancing LLM Reasoning Generalists with Preference Treescited by 191
  15. 2023Does Writing with Language Models Reduce Content Diversity?cited by 190
  16. 2021GPT3Mix: Leveraging Large-scale Language Models for Text Augmentationcited by 185
  17. 2018Online normalizer calculation for softmaxcited by 154
  18. 2023AugGPT: Leveraging ChatGPT for Text Data Augmentationcited by 128
  19. 2020Evaluating the Evaluation of Diversity in Natural Language Generationcited by 127
  20. 2023Increasing Diversity While Maintaining Accuracy: Text Data Generation with Large Language Models and Human Interventionscited by 121
  21. 2017Natural Language Processing: State of The Art, Current Trends and Challengescited by 114
  22. 2017Deep representation learning for human motion prediction and classificationcited by 97
  23. 2018Language GANs Falling Shortcited by 97
  24. 2023Instruction Mining: Instruction Data Selection for Tuning Large Language Modelscited by 57
  25. 2020Diverse Image Generation via Self-Conditioned GANscited by 54
  26. 2020Affinity and Diversity: Quantifying Mechanisms of Data Augmentationcited by 45
  27. 2024Synthetic Dialogue Dataset Generation using LLM Agentscited by 39
  28. 2021GradDiv: Adversarial Robustness of Randomized Neural Networks via Gradient Diversity Regularizationcited by 34
  29. 2023Large Language Models for Healthcare Data Augmentation: An Example on Patient-Trial Matchingcited by 30
  30. 2022Semantic Diversity in Dialogue with Natural Language Inferencecited by 28
  31. 2024Exploring Precision and Recall to assess the quality and diversity of LLMscited by 27
  32. 2024Scaling Data Diversity for Fine-Tuning Language Models in Human Alignmentcited by 25
  33. 2024Towards a Scalable Reference-Free Evaluation of Generative Modelscited by 25
  34. 2020DQI: Measuring Data Quality in NLPcited by 23
  35. 2023Mixture of Soft Prompts for Controllable Data Generationcited by 23
  36. 2020Diversity, Density, and Homogeneity: Quantitative Characteristic Metrics for Text Collectionscited by 21
  37. 2024Improving Diversity of Commonsense Generation by Large Language Models via In-Context Learningcited by 19
  38. 2024Empowering Large Language Models for Textual Data Augmentationcited by 17

50 most cited of 58 All 58 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 2; 2026: 3.

5 papers.

  1. 2026A Survey on Evaluating Quality and Trustworthiness in LLM-Generated Datacited by 5
  2. 2025SynQuE: Estimating Synthetic Dataset Quality Without Annotationscited by 1
  3. 2026Exploring More to Solve More: Boosting Diversity in Text Diffusion Models via Entropy-Based Guidancecited by 1
  4. 2025HAMMER: Hamiltonian Curiosity Augmented Large Language Model Reinforcementcited by 0
  5. 2026Measuring Dataset Diversity from a Geometric Perspectivecited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024On the Diversity of Synthetic Data and its Impact on Training Large Language Models
  2. 2024Efficacy of Synthetic Data as a Benchmark
  3. 2025Scaling Laws of Synthetic Data for Language Models
  4. 2025What Matters in LLM-generated Data: Diversity and Its Effect on Model Fine-Tuning
  5. 2025Demystifying Synthetic Data in LLM Pre-training: A Systematic Study of Scaling Laws, Benefits, and Pitfalls
  6. 2025Multi-Sample Prompting and Actor-Critic Prompt Optimization for Diverse Synthetic Data Generation
  7. 2025Data Value in the Age of Scaling: Understanding LLM Scaling Dynamics Under Real-Synthetic Data Mixtures
  8. 2023Generating Faithful Synthetic Data with Large Language Models: A Case Study in Computational Social Science