mlsanityData to 14 Oct 2026

Standardizing the Measurement of Text Diversity: A Tool and a Comparative Analysis of Scores

arXiv 2403.00553 ↗First posted 1 Mar 2024cs.CLcited by 50

Not on the map. Why?

Abstract (arXiv)

The diversity across outputs generated by LLMs shapes perception of their quality and utility. High lexical diversity is often desirable, but there is no standard method to measure this property. Templated answer structures and ``canned'' responses across different documents are readily noticeable, but difficult to visualize across large corpora. This work aims to standardize measurement of text diversity. Specifically, we empirically investigate the convergent validity of existing scores across English texts, and we release diversity, an open-source Python package for measuring and extracting repetition in text. We also build a platform based on diversity for users to interactively explore repetition in text. We find that fast compression algorithms capture information similar to what is measured by slow-to-compute n-gram overlap homogeneity scores. Further, a combination of measures -- compression ratios, self-repetition of long n-grams, and Self-BLEU and BERTScore -- are sufficient to report, as they have low mutual correlation with each other.

Key facts. Cited by 50 papers in the map (data through 14 Oct 2026). Builds on 23 of its 53 references that are in the map. First posted on arXiv 1 Mar 2024.

Builds on

23 of 53 references.

References · 23

  1. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 14,132
  2. 2019The Curious Case of Neural Text Degenerationcited by 2,916
  3. 2022Self-Instruct: Aligning Language Models with Self-Generated Instructionscited by 2,724
  4. 2023G-Eval: NLG Evaluation using GPT-4 with Better Human Alignmentcited by 2,276
  5. 2018Don't Give Me the Details, Just the Summary! Topic-Aware Convolutional Neural Networks for Extreme Summarizationcited by 1,317
  6. 2015A Diversity-Promoting Objective Function for Neural Conversation Modelscited by 1,077
  7. 2015Teaching Machines to Read and Comprehendcited by 1,049
  8. 2022Super-NaturalInstructions: Generalization via Declarative Instructions on 1600+ NLP Taskscited by 868
  9. 2023The Flan Collection: Designing Data and Methods for Effective Instruction Tuningcited by 739
  10. 2023OpenAssistant Conversations -- Democratizing Large Language Model Alignmentcited by 721
  11. 2021Cross-Task Generalization via Natural Language Crowdsourcing Instructionscited by 699
  12. 2022Contrastive Decoding: Open-ended Text Generation as Optimizationcited by 609
11 more
  1. 2018Texygen: A Benchmarking Platform for Text Generation Modelscited by 491
  2. 2022Unnatural Instructions: Tuning Language Models with (Almost) No Human Laborcited by 397
  3. 2023Does Writing with Language Models Reduce Content Diversity?cited by 192
  4. 2020Evaluating the Evaluation of Diversity in Natural Language Generationcited by 129
  5. 2022Understanding Factual Errors in Summarization: Errors, Summarizers, Datasets, Error Detectorscited by 104
  6. 2023The Curious Decline of Linguistic Diversity: Training Language Models on Synthetic Textcited by 100
  7. 2022On Decoding Strategies for Neural Text Generatorscited by 90
  8. 2021Directed Diversity: Leveraging Language Embedding Distances for Collective Creativity in Crowd Ideationcited by 23
  9. 2015Language Models for Image Captioning: The Quirks and What Workscited by 19
  10. 2023Automated Metrics for Medical Multi-Document Summarization Disagree with Human Evaluationscited by 18
  11. 2022Self-Repetition in Abstractive Neural Summarizerscited by 13

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 13; 2025: 30; 2026: 7.

50 papers.

Sort:

By area:

  1. 2024On the Diversity of Synthetic Data and its Impact on Training Large Language Modelscited by 59
  2. 2025Diverse Preference Optimizationcited by 52
  3. 2025Evaluating the Diversity and Quality of LLM Generated Contentcited by 51
  4. 2024Echoes in AI: Quantifying lack of plot diversity in LLM outputscited by 48
  5. 2024Detection and Measurement of Syntactic Templates in Generated Textcited by 44
  6. 2025Modifying Large Language Model Post-Training for Diverse Creative Writingcited by 40
  7. 2025Consistently Simulating Human Personas with Multi-Turn Reinforcement Learningcited by 39
  8. 2024From Distributional to Overton Pluralism: Investigating Large Language Model Alignmentcited by 33
  9. 2024Towards a Scalable Reference-Free Evaluation of Generative Modelscited by 26
  10. 2024Evaluating n-Gram Novelty of Language Models Using Rusty-DAWGcited by 25
  11. 2024Scaling Parameter-Constrained Language Models with Quality Datacited by 12
  12. 2025Diversity-driven Data Selection for Language Model Tuning through Sparse Autoencodercited by 12
38 more
  1. 2025LLM as a Broken Telephone: Iterative Generation Distorts Informationcited by 12
  2. 2024A SMART Mnemonic Sounds like "Glue Tonic": Mixing LLMs with Student Feedback to Make Mnemonic Learning Stickcited by 10
  3. 2025MetaSynth: Meta-Prompting-Driven Agentic Scaffolds for Diverse Synthetic Data Generationcited by 10
  4. 2025Improving Language Model Personas via Rationalization with Psychological Scaffoldscited by 9
  5. 2025Diverse, not Short: A Length-Controlled Data Selection Strategy for Improving Response Diversity of Language Modelscited by 9
  6. 2025Diversity-Enhanced Reasoning for Subjective Questionscited by 9
  7. 2025Who Taught You That? Tracing Teachers in Model Distillationcited by 8
  8. 2024DiVERT: Distractor Generation with Variational Errors Represented as Text for Math Multiple-choice Questionscited by 7
  9. 2025Measuring AI "Slop" in Textcited by 6
  10. 2025NavComposer: Composing Language Instructions for Navigation Trajectories through Action-Scene-Object Modularizationcited by 5
  11. 2025The Art of Asking: Multilingual Prompt Optimization for Synthetic Datacited by 5
  12. 2024QAEA-DR: A Unified Text Augmentation Framework for Dense Retrievalcited by 4
  13. 2024Utilizing Large Language Models to Synthesize Product Desirability Datasetscited by 4
  14. 2025Multi-Novelty: Improve the Diversity and Novelty of Contents Generated by Large Language Models via inference-time Multi-Views Brainstormingcited by 4
  15. 2025Can Large Language Models Really Recognize Your Name?cited by 4
  16. 2025RONA: Pragmatically Diverse Image Captioning with Coherence Relationscited by 3
  17. 2025Exploring Language Patterns of Prompts in Text-to-Image Generation and Their Impact on Visual Diversitycited by 3
  18. 2025From Macro to Micro: Probing Dataset Diversity in Language Model Fine-Tuningcited by 3
  19. 2025AutoSpec: An Agentic Framework for Automatically Drafting Patent Specificationcited by 3
  20. 2025Benchmarking Critical Questions Generation: A Challenging Reasoning Task for Large Language Modelscited by 2
  21. 2025We Need to Measure Data Diversity in NLP -- Better and Broadercited by 2
  22. 2025Do Language Models Think Consistently? A Study of Value Preferences Across Varying Response Lengthscited by 2
  23. 2026Who Wrote the Book? Detecting and Attributing LLM Ghostwriterscited by 2
  24. 2024Preserving Knowledge in Large Language Model with Model-Agnostic Self-Decompressioncited by 1
  25. 2024DiffSLT: Enhancing Diversity in Sign Language Translation via Diffusion Modelcited by 1
  26. 2025Thinking Outside the (Gray) Box: A Context-Based Score for Assessing Value and Originality in Neural Text Generationcited by 1
  27. 2025Privacy-Preserving Synthetic Review Generation with Diverse Writing Styles Using LLMscited by 1
  28. 2025How Persuasive Could LLMs Be? A First Study Combining Linguistic-Rhetorical Analysis and User Experimentscited by 1
  29. 2025Differentially-private text generation degrades output language qualitycited by 1
  30. 2025Robust Agents in Open-Ended Worldscited by 1
  31. 2026The Limits of Automatic Evaluation of Creativity in Large Language Modelscited by 1
  32. 2025A Penalty Goes a Long Way: Measuring Lexical Diversity in Synthetic Texts Under Prompt-Influenced Length Variationscited by 0
  33. 2025Measure what Matters: Psychometric Evaluation of AI with Situational Judgment Testscited by 0
  34. 2026SPA: A Simple but Tough-to-Beat Baseline for Knowledge Injectioncited by 0
  35. 2026Activation Steering for Synthetic Data Generation: The Role of Diversity in Downstream Safety Detectioncited by 0
  36. 2026Characterizing Cultural Localization in AI-Generated Storiescited by 0
  37. 2026MultiGhostBench: A Multilingual Benchmark for Long-Form LLM-Generated Text Attribution under Distribution Shiftscited by 0
  38. 2026Effective Synthetic Data Curation Requires Group-Level Signalscited by 0