mlsanityData to 7 Oct 2026

Evaluating the Evaluation of Diversity in Natural Language Generation

arXiv 2004.02990 ↗First posted 6 Apr 2020cs.CLcited by 127

Not on the map. Why?

Abstract (arXiv)

Despite growing interest in natural language generation (NLG) models that produce diverse outputs, there is currently no principled method for evaluating the diversity of an NLG system. In this work, we propose a framework for evaluating diversity metrics. The framework measures the correlation between a proposed diversity metric and a diversity parameter, a single parameter that controls some aspect of diversity in generated text. For example, a diversity parameter might be a binary variable used to instruct crowdsourcing workers to generate text with either low or high content diversity. We demonstrate the utility of our framework by: (a) establishing best practices for eliciting diversity judgments from humans, (b) showing that humans substantially outperform automatic metrics in estimating content diversity, and (c) demonstrating that existing methods for controlling diversity by tuning a "decoding parameter" mostly affect form but not meaning. Our framework can advance the understanding of different diversity metrics, an essential step on the road towards better NLG systems.

Key facts. Cited by 127 papers in the map (data through 7 Oct 2026). First posted on arXiv 6 Apr 2020.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 3; 2022: 8; 2023: 20; 2024: 27; 2025: 36; 2026: 33.

127 papers.

Sort:

By area:

  1. 2023Understanding the Effects of RLHF on LLM Generalisation and Diversitycited by 386
  2. 2023Large Language Model as Attributed Training Data Generator: A Tale of Diversity and Biascited by 230
  3. 2023Does Writing with Language Models Reduce Content Diversity?cited by 190
  4. 2024COLD-Attack: Jailbreaking LLMs with Stealthiness and Controllabilitycited by 162
  5. 2023The Curious Decline of Linguistic Diversity: Training Language Models on Synthetic Textcited by 97
  6. 2024Curiosity-driven Red-teaming for Large Language Modelscited by 83
  7. 2023Negative Object Presence Evaluation (NOPE) to Measure Object Hallucination in Vision-Language Modelscited by 80
  8. 2024LLMs know their vulnerabilities: Uncover Safety Gaps through Natural Distribution Shiftscited by 80
  9. 2024Evaluating Large Language Model Biases in Persona-Steered Generationcited by 78
  10. 2025NoveltyBench: Evaluating Language Models for Humanlike Diversitycited by 64
  11. 2024One fish, two fish, but not the whole sea: Alignment reduces language models' conceptual diversitycited by 57
  12. 2023What Comes Next? Evaluating Uncertainty in Neural Text Generators Against Human Production Variabilitycited by 51
38 more
  1. 2025Evaluating the Diversity and Quality of LLM Generated Contentcited by 51
  2. 2024Standardizing the Measurement of Text Diversity: A Tool and a Comparative Analysis of Scorescited by 50
  3. 2025Flipping the Dialogue: Training and Evaluating User Language Modelscited by 50
  4. 2024Forcing Diffuse Distributions out of Language Modelscited by 48
  5. 2025Diverse Preference Optimizationcited by 48
  6. 2025Massive Values in Self-Attention Modules are the Key to Contextual Knowledge Understandingcited by 44
  7. 2025The Price of Format: Diversity Collapse in LLMscited by 43
  8. 2025Modifying Large Language Model Post-Training for Diverse Creative Writingcited by 39
  9. 2024Benchmarking Language Model Creativity: A Case Study on Code Generationcited by 38
  10. 2024Benchmarking Linguistic Diversity of Large Language Modelscited by 37
  11. 2025The Homogenizing Effect of Large Language Models on Human Expression and Thoughtcited by 35
  12. 2024From Distributional to Overton Pluralism: Investigating Large Language Model Alignmentcited by 33
  13. 2024On the Challenges and Opportunities in Generative AIcited by 31
  14. 2022CTRLEval: An Unsupervised Reference-Free Metric for Evaluating Controlled Text Generationcited by 29
  15. 2024Unleashing the Power of Data Tsunami: A Comprehensive Survey on Data Assessment and Selection for Instruction Tuning of Language Modelscited by 29
  16. 2022Semantic Diversity in Dialogue with Natural Language Inferencecited by 28
  17. 2024Can LLMs Reason with Rules? Logic Scaffolding for Stress-Testing and Improving LLMscited by 28
  18. 2025LLM Probability Concentration: How Alignment Shrinks the Generative Horizoncited by 26
  19. 2025Multilingual Prompting for Improving LLM Generation Diversitycited by 25
  20. 2025Measuring Data Diversity for Instruction Tuning: A Systematic Analysis and A Reliable Metriccited by 24
  21. 2024Progress or Regress? Self-Improvement Reversal in Post-trainingcited by 23
  22. 2023How Far Can We Extract Diverse Perspectives from Large Language Models?cited by 22
  23. 2023Optimizing watermarks for large language modelscited by 22
  24. 2022Diversifying Content Generation for Commonsense Reasoning with Mixture of Knowledge Graph Expertscited by 19
  25. 2024Improving Diversity of Commonsense Generation by Large Language Models via In-Context Learningcited by 19
  26. 2024Do Large Language Models Have an English Accent? Evaluating and Improving the Naturalness of Multilingual LLMscited by 19
  27. 2023Vision-Language Instruction Tuning: A Review and Analysiscited by 17
  28. 2024Evaluating Diversity in Automatic Poetry Generationcited by 17
  29. 2022Measuring and Improving Semantic Diversity of Dialogue Generationcited by 16
  30. 2023A Systematic Study of Knowledge Distillation for Natural Language Generation with Pseudo-Target Trainingcited by 16
  31. 2023Metric Space Magnitude for Evaluating the Diversity of Latent Representationscited by 16
  32. 2026RAGEN-2: Reasoning Collapse in Agentic RLcited by 16
  33. 2024LLM-based multi-agent poetry generation in non-cooperative environmentscited by 15
  34. 2023CoAScore: Chain-of-Aspects Prompting for NLG Evaluationcited by 14
  35. 2025Curiosity-Driven Reinforcement Learning from Human Feedbackcited by 14
  36. 2022Paraphrase Identification with Deep Learning: A Review of Datasets and Methodscited by 13
  37. 2024Toward Optimal LLM Alignments Using Two-Player Gamescited by 13
  38. 2024Generating Diverse and High-Quality Texts by Minimum Bayes Risk Decodingcited by 11

50 most cited of 127 All 127 (JSON)