mlsanityData to 14 Oct 2026

Benchmarking Linguistic Diversity of Large Language Models

arXiv 2412.10271 ↗First posted 13 Dec 2024cs.CLcited by 38

Not on the map. Why?

Abstract (arXiv)

The development and evaluation of Large Language Models (LLMs) has primarily focused on their task-solving capabilities, with recent models even surpassing human performance in some areas. However, this focus often neglects whether machine-generated language matches the human level of diversity, in terms of vocabulary choice, syntactic construction, and expression of meaning, raising questions about whether the fundamentals of language generation have been fully addressed. This paper emphasizes the importance of examining the preservation of human linguistic richness by language models, given the concerning surge in online content produced or aided by LLMs. We propose a comprehensive framework for evaluating LLMs from various linguistic diversity perspectives including lexical, syntactic, and semantic dimensions. Using this framework, we benchmark several state-of-the-art LLMs across all diversity dimensions, and conduct an in-depth case study for syntactic diversity. Finally, we analyze how different development and deployment choices impact the linguistic diversity of LLM outputs.

Key facts. Cited by 38 papers in the map (data through 14 Oct 2026). Builds on 57 of its 81 references that are in the map. First posted on arXiv 13 Dec 2024.

Builds on

57 of 81 references.

References · 57

  1. 2020Language Models are Few-Shot Learnerscited by 32,419
  2. 2024The Llama 3 Herd of Modelscited by 14,715
  3. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 13,028
  4. 2019Sentence-BERT: Sentence Embeddings using Siamese BERT-Networkscited by 9,170
  5. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 8,065
  6. 2020Measuring Massive Multitask Language Understandingcited by 7,963
  7. 2019BERTScore: Evaluating Text Generation with BERTcited by 5,512
  8. 2022OPT: Open Pre-trained Transformer Language Modelscited by 3,512
  9. 2023Mistral 7Bcited by 3,164
  10. 2016Pointer Sentinel Mixture Modelscited by 3,096
  11. 2019The Curious Case of Neural Text Degenerationcited by 2,916
  12. 2023G-Eval: NLG Evaluation using GPT-4 with Better Human Alignmentcited by 2,276
38 more
  1. 2024Qwen2 Technical Reportcited by 2,120
  2. 2024Gemma 2: Improving Open Language Models at a Practical Sizecited by 1,998
  3. 2024MMLU-Pro: A More Robust and Challenging Multi-Task Language Understanding Benchmarkcited by 1,848
  4. 2018Hierarchical Neural Story Generationcited by 1,193
  5. 2020COMET: A Neural Framework for MT Evaluationcited by 1,033
  6. 2020On Faithfulness and Factuality in Abstractive Summarizationcited by 900
  7. 2023Quantifying Language Models' Sensitivity to Spurious Features in Prompt Design or: How I learned to start worrying about prompt formattingcited by 817
  8. 2020Stanza: A Python Natural Language Processing Toolkit for Many Human Languagescited by 723
  9. 2023Bias and Fairness in Large Language Models: A Surveycited by 639
  10. 2024OLMo: Accelerating the Science of Language Modelscited by 613
  11. 2020Making Monolingual Sentence Embeddings Multilingual using Knowledge Distillationcited by 593
  12. 2023The Falcon Series of Open Language Modelscited by 535
  13. 2024Dolma: an Open Corpus of Three Trillion Tokens for Language Model Pretraining Researchcited by 483
  14. 2021MAUVE: Measuring the Gap Between Neural Text and Human Text using Divergence Frontierscited by 470
  15. 2022Large Language Models Struggle to Learn Long-Tail Knowledgecited by 458
  16. 2023UltraFeedback: Boosting Language Models with Scaled AI Feedbackcited by 406
  17. 2023Understanding the Effects of RLHF on LLM Generalisation and Diversitycited by 396
  18. 2021XL-Sum: Large-Scale Multilingual Abstractive Summarization for 44 Languagescited by 347
  19. 2023Camels in a Changing Climate: Enhancing LM Adaptation with Tulu 2cited by 229
  20. 2023SafetyBench: Evaluating the Safety of Large Language Modelscited by 199
  21. 2023Does Writing with Language Models Reduce Content Diversity?cited by 192
  22. 2020Evaluating the Evaluation of Diversity in Natural Language Generationcited by 129
  23. 2024Mapping the Increasing Use of LLMs in Scientific Paperscited by 118
  24. 2023The Curious Decline of Linguistic Diversity: Training Language Models on Synthetic Textcited by 100
  25. 2018Language GANs Falling Shortcited by 97
  26. 2020Trading Off Diversity and Quality in Natural Language Generationcited by 96
  27. 2022Of Human Criteria and Automatic Metrics: A Benchmark of the Evaluation of Story Generationcited by 75
  28. 2022Help me write a poem: Instruction Tuning as a Vehicle for Collaborative Poetry Writingcited by 71
  29. 2020Improving Dialog Evaluation with a Multi-reference Adversarial Dataset and Large Scale Pretrainingcited by 59
  30. 2018GraKeL: A Graph Kernel Library in Pythoncited by 54
  31. 2023What Comes Next? Evaluating Uncertainty in Neural Text Generators Against Human Production Variabilitycited by 51
  32. 2023Improving Diversity of Demographic Representation in Large Language Models via Collective-Critiques and Self-Votingcited by 46
  33. 2024Detection and Measurement of Syntactic Templates in Generated Textcited by 44
  34. 2024Few-Shot Detection of Machine-Generated Text using Style Representationscited by 40
  35. 2022Semantic Diversity in Dialogue with Natural Language Inferencecited by 29
  36. 2024Exploring Precision and Recall to assess the quality and diversity of LLMscited by 29
  37. 2024Is ChatGPT Transforming Academics' Writing Style?cited by 26
  38. 2017Hierarchical Text Generation and Planning for Strategic Dialoguecited by 23

50 most cited of 57 All 57 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 15; 2026: 23.

38 papers.

Sort:

By area:

  1. 2025Do Multilingual LLMs Think In English?cited by 78
  2. 2025Diverse Preference Optimizationcited by 52
  3. 2025Evaluating the Diversity and Quality of LLM Generated Contentcited by 51
  4. 2025The Homogenizing Effect of Large Language Models on Human Expression and Thoughtcited by 35
  5. 2025LLM Probability Concentration: How Alignment Shrinks the Generative Horizoncited by 26
  6. 2025Post-training Large Language Models for Diverse High-Quality Responsescited by 19
  7. 2025What and Whose Knowledge? Measuring Epistemic Diversity in Large Language Modelscited by 14
  8. 2025Detecting Stylistic Fingerprints of Large Language Modelscited by 11
  9. 2026The Roots of Performance Disparity in Multilingual Language Models: Intrinsic Modeling Difficulty or Design Choices?cited by 11
  10. 2026Where does output diversity collapse in post-training?cited by 11
  11. 2025Deterministic or probabilistic? The psychology of LLMs as random number generatorscited by 7
  12. 2026ALRM: Agentic LLM for Robotic Manipulationcited by 6
26 more
  1. 2025A survey of diversity quantification in natural language processing: The why, what, where and howcited by 4
  2. 2026Markovian Generation Chains in Large Language Modelscited by 3
  3. 2025QFrBLiMP: a Quebec-French Benchmark of Linguistic Minimal Pairscited by 2
  4. 2025Generation Space Size: Understanding and Calibrating Open-Endedness of LLM Generationscited by 2
  5. 2026The Homogenization Problem in LLMs: Towards Meaningful Diversity in AI Safetycited by 2
  6. 2026Sampling More, Getting Less: Calibration is the Diversity Bottleneck in LLMscited by 2
  7. 2025BiasGym: A Simple and Generalizable Framework for Analyzing and Removing Biases through Injectioncited by 1
  8. 2025AI Brown and AI Koditex: LLM-Generated Corpora Comparable to Traditional Corpora of English and Czech Textscited by 1
  9. 2025Unveiling Intrinsic Dimension of Texts: from Academic Abstract to Creative Storycited by 1
  10. 2026Quality-constrained Entropy Maximization Policy Optimization for LLM Diversitycited by 1
  11. 2026Incentives shape how humans co-create with generative AIcited by 1
  12. 2026Evaluating the Realism of LLM-powered Social Agents: A Case Study of Reactions to Spanish Online Newscited by 1
  13. 2026Is Convergence Inevitable? Tracing Output Homogeneity Back to Base Modelscited by 1
  14. 2026Forty Shades of Blue: Quality-Diversity Alignment via Mode-Conditioned Reinforcement Learningcited by 1
  15. 2026Does AI Homogenize Student Thinking? A Multi-Dimensional Analysis of Structural Convergence in AI-Augmented Essayscited by 0
  16. 2026Magic, Madness, Heaven, Sin: LLM Output Diversity is Everything, Everywhere, All at Oncecited by 0
  17. 2026The TEA Nets framework combines AI and cognitive network science to model targets, events and actors in textcited by 0
  18. 2026CommonWhy: A Dataset for Evaluating Entity-Based Causal Commonsense Reasoning in Large Language Modelscited by 0
  19. 2026Fine-Tuning Improves Information Conveyance in Language Modelscited by 0
  20. 2026"I've Seen How This Goes": Characterizing Diversity via Progressive Conditional Surprisecited by 0
  21. 2026Behavioral and Representational Evidence of Binomial Ordering Preferences in Large Language Modelscited by 0
  22. 2026Refusal-Gated Decoding: Preserving Refusal Behavior Under High-Temperature Samplingcited by 0
  23. 2026Evaluating Multiple LLM Generations with Validated Task Coveragecited by 0
  24. 2026Beyond Fluency: A Rubric-Based Benchmark for Evaluating Saudi Dialect and Cultural Competence in Large Language Modelscited by 0
  25. 2026On Epistemic Diversity in Large Language Modelscited by 0
  26. 2026Explicit Trajectory Diversity for RL-Based Post-Training of LLM Agentscited by 0