mlsanityData to 7 Oct 2026

The Curious Decline of Linguistic Diversity: Training Language Models on Synthetic Text

arXiv 2311.09807 ↗First posted 16 Nov 2023cs.CLcited by 97

Not on the map. Why?

Cited by 27 in Synthetic data and self-improvement for LLMs, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

This study investigates the consequences of training language models on synthetic data generated by their predecessors, an increasingly prevalent practice given the prominence of powerful generative models. Diverging from the usual emphasis on performance metrics, we focus on the impact of this training methodology on linguistic diversity, especially when conducted recursively over time. To assess this, we adapt and develop a set of novel metrics targeting lexical, syntactic, and semantic diversity, applying them in recursive finetuning experiments across various natural language generation tasks in English. Our findings reveal a consistent decrease in the diversity of the model outputs through successive iterations, especially remarkable for tasks demanding high levels of creativity. This trend underscores the potential risks of training language models on synthetic text, particularly concerning the preservation of linguistic richness. Our study highlights the need for careful consideration of the long-term effects of such training approaches on the linguistic capabilities of language models.

Key facts. Cited by 97 papers in the map (data through 7 Oct 2026). Builds on 43 of its 71 references that are in the map. First posted on arXiv 16 Nov 2023.

Builds on

43 of 71 references.

References · 43

  1. 2022Training language models to follow instructions with human feedbackcited by 15,712
  2. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  3. 2019Sentence-BERT: Sentence Embeddings using Siamese BERT-Networkscited by 9,049
  4. 2022Self-Consistency Improves Chain of Thought Reasoning in Language Modelscited by 5,601
  5. 2022OPT: Open Pre-trained Transformer Language Modelscited by 3,464
  6. 2019The Curious Case of Neural Text Degenerationcited by 2,881
  7. 2022Self-Instruct: Aligning Language Models with Self-Generated Instructionscited by 2,674
  8. 2018Don't Give Me the Details, Just the Summary! Topic-Aware Convolutional Neural Networks for Extreme Summarizationcited by 1,308
  9. 2018Hierarchical Neural Story Generationcited by 1,182
  10. 2015A Diversity-Promoting Objective Function for Neural Conversation Modelscited by 1,068
  11. 2015Teaching Machines to Read and Comprehendcited by 1,039
  12. 2018Neural Network Acceptability Judgmentscited by 952
31 more
  1. 2020Stanza: A Python Natural Language Processing Toolkit for Many Human Languagescited by 719
  2. 2022Large Language Models Can Self-Improvecited by 656
  3. 2022Contrastive Decoding: Open-ended Text Generation as Optimizationcited by 598
  4. 2018Texygen: A Benchmarking Platform for Text Generation Modelscited by 484
  5. 2021XL-Sum: Large-Scale Multilingual Abstractive Summarization for 44 Languagescited by 346
  6. 2023The Curse of Recursion: Training on Generated Data Makes Models Forgetcited by 343
  7. 2023Self-Consuming Generative Models Go MADcited by 239
  8. 2022Predictability and Surprise in Large Generative Modelscited by 209
  9. 2023Does Writing with Language Models Reduce Content Diversity?cited by 190
  10. 2022Will we run out of data? Limits of LLM scaling based on human-generated datacited by 187
  11. 2023Art or Artifice? Large Language Models and the False Promise of Creativitycited by 168
  12. 2020Evaluating the Evaluation of Diversity in Natural Language Generationcited by 127
  13. 2021How much do language models copy from their training data? Evaluating linguistic novelty in text generation using RAVENcited by 124
  14. 2016Neural Summarization by Extracting Sentences and Wordscited by 114
  15. 2020Trading Off Diversity and Quality in Natural Language Generationcited by 94
  16. 2021Data Augmentation Approaches in Natural Language Processing: A Surveycited by 82
  17. 2019Comparison of Diverse Decoding Methods from Conditional Language Modelscited by 78
  18. 2022Help me write a poem: Instruction Tuning as a Vehicle for Collaborative Poetry Writingcited by 71
  19. 2019Jointly Measuring Diversity and Quality in Text Generation Modelscited by 61
  20. 2018GraKeL: A Graph Kernel Library in Pythoncited by 54
  21. 2023What Comes Next? Evaluating Uncertainty in Neural Text Generators Against Human Production Variabilitycited by 51
  22. 2022Semantic Diversity in Dialogue with Natural Language Inferencecited by 28
  23. 2017Hierarchical Text Generation and Planning for Strategic Dialoguecited by 23
  24. 2022AmbiPun: Generating Humorous Puns with Ambiguous Contextcited by 21
  25. 2023Does Human Collaboration Enhance the Accuracy of Identifying LLM-Generated Deepfake Texts?cited by 21
  26. 2022Measuring and Improving Semantic Diversity of Dialogue Generationcited by 16
  27. 2023ParaAMR: A Large-Scale Syntactically Diverse Paraphrase Dataset by AMR Back-Translationcited by 11
  28. 2022Questioning the Validity of Summarization Datasets and Improving Their Factual Consistencycited by 9
  29. 2022Towards Robust NLG Bias Evaluation with Syntactically-diverse Promptscited by 7
  30. 2023Multi-Stage Pre-training Enhanced by ChatGPT for Multi-Scenario Multi-Domain Dialogue Summarizationcited by 7
  31. 2023The Use of Synthetic Data to Train AI Models: Opportunities and Risks for Sustainable Developmentcited by 6

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 1; 2024: 26; 2025: 36; 2026: 34.

97 papers.

Sort:

By area:

  1. 2024A Tale of Tails: Model Collapse as a Change of Scaling Lawscited by 127
  2. 2024A Survey on Self-Evolution of Large Language Modelscited by 86
  3. 2023Large Language Models Suffer From Their Own Output: An Analysis of the Self-Consuming Training Loopcited by 75
  4. 2024Preserving Diversity in Supervised Fine-Tuning of Large Language Modelscited by 68
  5. 2025SE-Agent: Self-Evolution Trajectory Optimization in Multi-Step Reasoning with LLM-Based Agentscited by 66
  6. 2024Standardizing the Measurement of Text Diversity: A Tool and a Comparative Analysis of Scorescited by 50
  7. 2024Collapse or Thrive? Perils and Promises of Synthetic Data in a Self-Generating Worldcited by 49
  8. 2024Benchmarking Linguistic Diversity of Large Language Modelscited by 37
  9. 2025The Homogenizing Effect of Large Language Models on Human Expression and Thoughtcited by 35
  10. 2024AI and the Problem of Knowledge Collapsecited by 34
  11. 2025The Shrinking Landscape of Linguistic Diversity in the Age of Large Language Modelscited by 28
  12. 2024Empirical evidence of Large Language Model's influence on human spoken communicationcited by 27
38 more
  1. 2024Do Large Language Models Have an English Accent? Evaluating and Improving the Naturalness of Multilingual LLMscited by 19
  2. 2025BeyondWeb: Lessons from Scaling Synthetic Data for Trillion-scale Pretrainingcited by 19
  3. 2024The Impact of Large Language Models in Academia: from Writing to Speakingcited by 18
  4. 2025Superficial Self-Improved Reasoners Benefit from Model Mergingcited by 16
  5. 2025Persona-Augmented Benchmarking: Evaluating LLMs Across Diverse Writing Stylescited by 15
  6. 2025Machine-generated text detection prevents language model collapsecited by 14
  7. 2024Atlas-Chat: Adapting Large Language Models for Low-Resource Moroccan Arabic Dialectcited by 13
  8. 2024Mitigating Tail Narrowing in LLM Self-Improvement via Socratic-Guided Samplingcited by 13
  9. 2024Bias Amplification: Large Language Models as Increasingly Biased Mediacited by 12
  10. 2025LLM as a Broken Telephone: Iterative Generation Distorts Informationcited by 12
  11. 2024Model Collapse in the Self-Consuming Chain of Diffusion Finetuning: A Novel Perspective from Quantitative Trait Modelingcited by 11
  12. 2024Importance Weighting Can Help Large Language Models Self-Improvecited by 11
  13. 2025When Models Don't Collapse: On the Consistency of Iterative MLEcited by 11
  14. 2025Your Language Model Can Secretly Write Like Humans: Contrastive Paraphrase Attacks on LLM-Generated Text Detectorscited by 10
  15. 2024Leveraging Synthetic Audio Data for End-to-End Low-Resource Speech Translationcited by 9
  16. 2025Diverse, not Short: A Length-Controlled Data Selection Strategy for Improving Response Diversity of Language Modelscited by 9
  17. 2024Multi-Agent Sampling: Scaling Inference Compute for Data Synthesis with Tree Search-Based Agentic Collaborationcited by 8
  18. 2025BARE: Leveraging Base Language Models for Few-Shot Synthetic Data Generationcited by 8
  19. 2025Why Less is More (Sometimes): A Theory of Data Curationcited by 8
  20. 2025Aligning Instruction Tuning with Pre-trainingcited by 7
  21. 2024LLM See, LLM Do: Guiding Data Generation to Target Non-Differentiable Objectivescited by 6
  22. 2024Multilingual Arbitrage: Optimizing Data Pools to Accelerate Multilingual Progresscited by 6
  23. 2024Maximizing the Potential of Synthetic Data: Insights from Random Matrix Theorycited by 6
  24. 2024A Little Human Data Goes A Long Waycited by 6
  25. 2025An Interdisciplinary Approach to Human-Centered Machine Translationcited by 6
  26. 2025Semantic Voting: A Self-Evaluation-Free Approach for Efficient LLM Self-Improvement on Unverifiable Open-ended Taskscited by 6
  27. 2024Benchmarking LLMs for Mimicking Child-Caregiver Language in Interactioncited by 5
  28. 2026Limited Linguistic Diversity in Embodied AI Datasetscited by 5
  29. 2025A survey of diversity quantification in natural language processing: The why, what, where and howcited by 4
  30. 2024Augmenting Compliance-Guaranteed Customer Service Chatbots: Context-Aware Knowledge Expansion with Large Language Modelscited by 3
  31. 2025Research Borderlands: Analysing Writing Across Research Culturescited by 3
  32. 2025Large Language Models as Search Engines: Societal Challengescited by 3
  33. 2026Language of Thought Shapes Output Diversity in Large Language Modelscited by 3
  34. 2026Markovian Generation Chains in Large Language Modelscited by 3
  35. 2026The One-Word Census: Answer-Choice Conformity Across 44 Language Modelscited by 3
  36. 2024Learning by Surprise: Adaptive Mitigation of Model Collapse in Large Language Modelscited by 2
  37. 2025Beyond Sample-Level Feedback: Using Reference-Level Feedback to Guide Data Synthesiscited by 2
  38. 2025Recursive Training Loops in LLMs: How training data properties modulate distribution shift in generated data?cited by 2

50 most cited of 97 All 97 (JSON)