# AI and the Problem of Knowledge Collapse

arXiv: https://arxiv.org/abs/2404.03502 · HTML: https://mlsanity.com/paper/2404.03502/

## Key facts

- Cited by 34 papers in the map (data through 14 Oct 2026).
- Builds on 28 of its 107 references that are in the map.
- First posted on arXiv 4 Apr 2024.

## Builds on (most cited first, 28 of 28)

- [Scaling Laws for Neural Language Models](https://mlsanity.com/paper/2001.08361/) (2020)
- [Training Compute-Optimal Large Language Models](https://mlsanity.com/paper/2203.15556/) (2022)
- [The Curious Case of Neural Text Degeneration](https://mlsanity.com/paper/1904.09751/) (2019)
- [Large Language Models Struggle to Learn Long-Tail Knowledge](https://mlsanity.com/paper/2211.08411/) (2022)
- [The Curse of Recursion: Training on Generated Data Makes Models Forget](https://mlsanity.com/paper/2305.17493/) (2023)
- [NIPS 2016 Tutorial: Generative Adversarial Networks](https://mlsanity.com/paper/1701.00160/) (2016)
- [Do Llamas Work in English? On the Latent Language of Multilingual Transformers](https://mlsanity.com/paper/2402.10588/) (2024)
- [Large Language Model as Attributed Training Data Generator: A Tale of Diversity and Bias](https://mlsanity.com/paper/2306.15895/) (2023)
- [A Contrastive Framework for Neural Text Generation](https://mlsanity.com/paper/2202.06417/) (2022)
- [Generalization and Equilibrium in Generative Adversarial Nets (GANs)](https://mlsanity.com/paper/1703.00573/) (2017)
- [A Tale of Tails: Model Collapse as a Change of Scaling Laws](https://mlsanity.com/paper/2402.07043/) (2024)
- [AtP*: An efficient and scalable method for localizing LLM behaviour to components](https://mlsanity.com/paper/2403.00745/) (2024)
- [Interpretability at Scale: Identifying Causal Mechanisms in Alpaca](https://mlsanity.com/paper/2305.08809/) (2023)
- [The Curious Decline of Linguistic Diversity: Training Language Models on Synthetic Text](https://mlsanity.com/paper/2311.09807/) (2023)
- [Combating Misinformation in the Age of LLMs: Opportunities and Challenges](https://mlsanity.com/paper/2311.05656/) (2023)
- [On Decoding Strategies for Neural Text Generators](https://mlsanity.com/paper/2203.15721/) (2022)
- [Degenerate Feedback Loops in Recommender Systems](https://mlsanity.com/paper/1902.10730/) (2019)
- [User-centered Evaluation of Popularity Bias in Recommender Systems](https://mlsanity.com/paper/2103.06364/) (2021)
- [CIRS: Bursting Filter Bubbles by Counterfactual Interactive Recommender System](https://mlsanity.com/paper/2204.01266/) (2022)
- [Nepotistically Trained Generative-AI Models Collapse](https://mlsanity.com/paper/2311.12202/) (2023)
- [Under the Surface: Tracking the Artifactuality of LLM-Generated Data](https://mlsanity.com/paper/2401.14698/) (2024)
- [The Effect of People Recommenders on Echo Chambers and Polarization](https://mlsanity.com/paper/2112.00626/) (2021)
- [Graph-Based Retriever Captures the Long Tail of Biomedical Knowledge](https://mlsanity.com/paper/2402.12352/) (2024)
- [Data Debugging with Shapley Importance over End-to-End Machine Learning Pipelines](https://mlsanity.com/paper/2204.11131/) (2022)
- [Generative Echo Chamber? Effects of LLM-Powered Search Systems on Diverse Information Seeking](https://mlsanity.com/paper/2402.05880/) (2024)
- [Quantifying and Mitigating Popularity Bias in Conversational Recommender Systems](https://mlsanity.com/paper/2208.03298/) (2022)
- [In Search of the Long-Tail: Systematic Generation of Long-Tail Inferential Knowledge via Logical Rule Guided Search](https://mlsanity.com/paper/2311.07237/) (2023)
- [Knowledge Base Completion for Long-Tail Entities](https://mlsanity.com/paper/2306.17472/) (2023)

## Built on by (most cited first, 30 of 34)

- [The 2025 AI Agent Index: Documenting Technical and Safety Features of Deployed Agentic AI Systems](https://mlsanity.com/paper/2602.17753/) (2026)
- [Do Large Language Models Have an English Accent? Evaluating and Improving the Naturalness of Multilingual LLMs](https://mlsanity.com/paper/2410.15956/) (2024)
- [The Impact of Large Language Models in Academia: from Writing to Speaking](https://mlsanity.com/paper/2409.13686/) (2024)
- [Can We Use Large Language Models to Fill Relevance Judgment Holes?](https://mlsanity.com/paper/2405.05600/) (2024)
- [When LLMs Play the Telephone Game: Cultural Attractors as Conceptual Tools to Evaluate LLMs in Multi-turn Settings](https://mlsanity.com/paper/2407.04503/) (2024)
- [What and Whose Knowledge? Measuring Epistemic Diversity in Large Language Models](https://mlsanity.com/paper/2510.04226/) (2025)
- [LLM as a Broken Telephone: Iterative Generation Distorts Information](https://mlsanity.com/paper/2502.20258/) (2025)
- [The Lock-in Hypothesis: Stagnation by Algorithm](https://mlsanity.com/paper/2506.06166/) (2025)
- [ProgressGym: Alignment with a Millennium of Moral Progress](https://mlsanity.com/paper/2406.20087/) (2024)
- [Variance reduction in output from generative AI](https://mlsanity.com/paper/2503.01033/) (2025)
- [What happens when generative AI models train recursively on each others' outputs?](https://mlsanity.com/paper/2505.21677/) (2025)
- [CrowdLLM: Building LLM-Based Digital Populations Augmented with Generative Models](https://mlsanity.com/paper/2512.07890/) (2025)
- [Causes and Consequences of Representational Similarity in Machine Learning Models](https://mlsanity.com/paper/2505.13899/) (2025)
- [Markovian Generation Chains in Large Language Models](https://mlsanity.com/paper/2603.11228/) (2026)
- [Convergence Dynamics and Stabilization Strategies of Co-Evolving Generative Models](https://mlsanity.com/paper/2503.08117/) (2025)
- [Epistemic diversity across language models mitigates knowledge collapse](https://mlsanity.com/paper/2512.15011/) (2025)
- [The Homogenization Problem in LLMs: Towards Meaningful Diversity in AI Safety](https://mlsanity.com/paper/2601.06116/) (2026)
- [AI in the Writing Process: How Purposeful AI Support Fosters Student Writing](https://mlsanity.com/paper/2506.20595/) (2025)
- [Knowledge Collapse in LLMs: When Fluency Survives but Facts Fail under Recursive Synthetic Training](https://mlsanity.com/paper/2509.04796/) (2025)
- [LLM in the Middle: A Systematic Review of Threats and Mitigations to Real-World LLM-based Systems](https://mlsanity.com/paper/2509.10682/) (2025)
- [Big AI is accelerating the metacrisis: What can we do?](https://mlsanity.com/paper/2512.24863/) (2025)
- [AI-Mediated Communication Can Steer Collective Opinion](https://mlsanity.com/paper/2605.16245/) (2026)
- [Argument Collapse: LLMs Flatten Long-Form Public Debate](https://mlsanity.com/paper/2606.01736/) (2026)
- [At the edge of a generative cultural precipice](https://mlsanity.com/paper/2406.08739/) (2024)
- [Training for Obsolescence? The AI-Driven Education Trap](https://mlsanity.com/paper/2508.19625/) (2025)
- [Future of AI Models: A Computational perspective on Model collapse](https://mlsanity.com/paper/2511.05535/) (2025)
- ["Oops! ChatGPT is Temporarily Unavailable!": A Diary Study on Knowledge Workers' Experiences of LLM Withdrawal](https://mlsanity.com/paper/2603.26099/) (2026)
- [Using AI in engineering education: a balancing act, driven by clear purpose](https://mlsanity.com/paper/2606.16626/) (2026)
- [Linguistic Monoculture in LLM-Assisted Language Use](https://mlsanity.com/paper/2607.27134/) (2026)
- [When AI Writes, Who Gets Cited? Evidence of Citation Monoculture Across Language Models](https://mlsanity.com/paper/2608.19230/) (2026)

## Abstract (arXiv)

While artificial intelligence has the potential to process vast amounts of data, generate new insights, and unlock greater productivity, its widespread adoption may entail unforeseen consequences. We identify conditions under which AI, by reducing the cost of access to certain modes of knowledge, can paradoxically harm public understanding. While large language models are trained on vast amounts of diverse data, they naturally generate output towards the 'center' of the distribution. This is generally useful, but widespread reliance on recursive AI systems could lead to a process we define as "knowledge collapse", and argue this could harm innovation and the richness of human understanding and culture. However, unlike AI models that cannot choose what data they are trained on, humans may strategically seek out diverse forms of knowledge if they perceive them to be worthwhile. To investigate this, we provide a simple model in which a community of learners or innovators choose to use traditional methods or to rely on a discounted AI-assisted process and identify conditions under which knowledge collapse occurs. In our default model, a 20% discount on AI-generated content generates public beliefs 2.3 times further from the truth than when there is no discount. An empirical approach to measuring the distribution of LLM outputs is provided in theoretical terms and illustrated through a specific example comparing the diversity of outputs across different models and prompting styles. Finally, based on the results, we consider further research directions to counteract such outcomes.