# Epistemic diversity across language models mitigates knowledge collapse

arXiv: https://arxiv.org/abs/2512.15011 · HTML: https://mlsanity.com/paper/2512.15011/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 32 of its 96 references that are in the map.
- First posted on arXiv 17 Dec 2025.

## Builds on (most cited first, 30 of 32)

- [Training language models to follow instructions with human feedback](https://mlsanity.com/paper/2203.02155/) (2022)
- [Scaling Laws for Neural Language Models](https://mlsanity.com/paper/2001.08361/) (2020)
- [OPT: Open Pre-trained Transformer Language Models](https://mlsanity.com/paper/2205.01068/) (2022)
- [Pointer Sentinel Mixture Models](https://mlsanity.com/paper/1609.07843/) (2016)
- [Constitutional AI: Harmlessness from AI Feedback](https://mlsanity.com/paper/2212.08073/) (2022)
- [HuggingFace's Transformers: State-of-the-art Natural Language Processing](https://mlsanity.com/paper/1910.03771/) (2019)
- [Datasheets for Datasets](https://mlsanity.com/paper/1803.09010/) (2018)
- [Model Cards for Model Reporting](https://mlsanity.com/paper/1810.03993/) (2018)
- [Beyond neural scaling laws: beating power law scaling via data pruning](https://mlsanity.com/paper/2206.14486/) (2022)
- [Data Selection for Language Models via Importance Resampling](https://mlsanity.com/paper/2302.03169/) (2023)
- [Is Model Collapse Inevitable? Breaking the Curse of Recursion by Accumulating Real and Synthetic Data](https://mlsanity.com/paper/2404.01413/) (2024)
- [A Tale of Tails: Model Collapse as a Change of Scaling Laws](https://mlsanity.com/paper/2402.07043/) (2024)
- [Increasing Diversity While Maintaining Accuracy: Text Data Generation with Large Language Models and Human Interventions](https://mlsanity.com/paper/2306.04140/) (2023)
- [Modular Pluralism: Pluralistic Alignment via Multi-LLM Collaboration](https://mlsanity.com/paper/2406.15951/) (2024)
- [Algorithmic Monoculture and Social Welfare](https://mlsanity.com/paper/2101.05853/) (2021)
- [On the Stability of Iterative Retraining of Generative Models on their own Data](https://mlsanity.com/paper/2310.00429/) (2023)
- [Aligner: Efficient Alignment by Learning to Correct](https://mlsanity.com/paper/2402.02416/) (2024)
- [Jointly Measuring Diversity and Quality in Text Generation Models](https://mlsanity.com/paper/1904.03971/) (2019)
- [Echoes in AI: Quantifying lack of plot diversity in LLM outputs](https://mlsanity.com/paper/2501.00273/) (2024)
- [Beyond Model Collapse: Scaling Up with Synthesized Data Requires Verification](https://mlsanity.com/paper/2406.07515/) (2024)
- [AI and the Problem of Knowledge Collapse](https://mlsanity.com/paper/2404.03502/) (2024)
- [Position: Model Collapse Does Not Mean What You Think](https://mlsanity.com/paper/2503.03150/) (2025)
- [A Theoretical Perspective: How to Prevent Model Collapse in Self-consuming Training Loops](https://mlsanity.com/paper/2502.18865/) (2025)
- [FisherSFT: Data-Efficient Supervised Fine-Tuning of Language Models Using Information Gain](https://mlsanity.com/paper/2505.14826/) (2025)
- [Selecting Informative Contexts Improves Language Model Finetuning](https://mlsanity.com/paper/2005.00175/) (2020)
- [Stop treating `AGI' as the north-star goal of AI research](https://mlsanity.com/paper/2502.03689/) (2025)
- [Escaping Collapse: The Strength of Weak Data for Large Language Model Training](https://mlsanity.com/paper/2502.08924/) (2025)
- [Political Neutrality in AI Is Impossible- But Here Is How to Approximate It](https://mlsanity.com/paper/2503.05728/) (2025)
- [Embracing Contradiction: Theoretical Inconsistency Will Not Impede the Road of Building Responsible AI Systems](https://mlsanity.com/paper/2505.18139/) (2025)
- [LLM Web Dynamics: Tracing Model Collapse in a Network of LLMs](https://mlsanity.com/paper/2506.15690/) (2025)

## Built on by (most cited first, 2 of 2)

- [Diversifying Personalized Research Ideation against AI-Induced Homogenization](https://mlsanity.com/paper/2607.28087/) (2026)
- [The Oligarch Barely Steers Model Collapse in Multi-Model Ecosystems](https://mlsanity.com/paper/2609.11146/) (2026)

## Abstract (arXiv)

Artificial intelligence (AI) increasingly generates the very content used to train future AI systems. This feedback loop can degrade model quality, reduce informational diversity, and ultimately drive knowledge collapse, i.e. a degradation to a narrow and inaccurate set of ideas. We ask: to mitigate collapse, is it better to concentrate the internet's knowledge into a handful of dominant models (referred to as an AI monoculture), or to distribute it across a diverse ecosystem of models? To study the effect of diversity on model performance, we randomly segment the fixed training data across an increasing number of language models and evaluate the resulting ecosystems of models over ten self-training iterations. Our results show that diversity improves long-term performance of models, while monoculture accelerates collapse. Specifically, we observe that the optimal diversity level (i.e., the level that maximizes performance) increases monotonically with the number of self-training iterations. The observed effect is robust across various experimental settings, including different model families, parameter sizes, mixing human- and model-generated data, and temperature sampling methods, demonstrating the significance of ecosystem diversity for mitigating collapse. Moreover, our experiments with increased model and dataset sizes indicate that scaling up the system can amplify collapse in homogeneous ecosystems, thereby increasing the diversity benefits. In the presence of AI monoculture, our results suggest considering information environments with specialized AI models that maintain and enhance diversity in knowledge production, akin to the benefits of ecological diversity in biology and social systems.