# LLM Web Dynamics: Tracing Model Collapse in a Network of LLMs

arXiv: https://arxiv.org/abs/2506.15690 · HTML: https://mlsanity.com/paper/2506.15690/

## Key facts

- Cited by 6 papers in the map (data through 14 Oct 2026).
- Placed in Model collapse from synthetic data, one of 13 directions in Synthetic data and self-improvement for LLMs.
- Builds on 24 of its 46 references that are in the map.
- First posted on arXiv 26 May 2025.

## Placed in

- Synthetic data and self-improvement for LLMs / [Model collapse from synthetic data](https://mlsanity.com/synth/model-collapse-from-synthetic-data/)

## Builds on (most cited first, 24 of 24)

- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [Scaling Laws for Neural Language Models](https://mlsanity.com/paper/2001.08361/) (2020)
- [Retrieval-Augmented Generation for Large Language Models: A Survey](https://mlsanity.com/paper/2312.10997/) (2023)
- [The Curse of Recursion: Training on Generated Data Makes Models Forget](https://mlsanity.com/paper/2305.17493/) (2023)
- [OpenChat: Advancing Open-source Language Models with Mixed-Quality Data](https://mlsanity.com/paper/2309.11235/) (2023)
- [Self-Consuming Generative Models Go MAD](https://mlsanity.com/paper/2307.01850/) (2023)
- [Agentic Retrieval-Augmented Generation: A Survey on Agentic RAG](https://mlsanity.com/paper/2501.09136/) (2025)
- [Will we run out of data? Limits of LLM scaling based on human-generated data](https://mlsanity.com/paper/2211.04325/) (2022)
- [Tensor Programs II: Neural Tangent Kernel for Any Architecture](https://mlsanity.com/paper/2006.14548/) (2020)
- [On the Turing Completeness of Modern Neural Network Architectures](https://mlsanity.com/paper/1901.03429/) (2019)
- [A Note on Shumailov et al. (2024): `AI Models Collapse When Trained on Recursively Generated Data'](https://mlsanity.com/paper/2410.12954/) (2024)
- [On the Stability of Iterative Retraining of Generative Models on their own Data](https://mlsanity.com/paper/2310.00429/) (2023)
- [RAG vs Fine-tuning: Pipelines, Tradeoffs, and a Case Study on Agriculture](https://mlsanity.com/paper/2401.08406/) (2024)
- [How Bad is Training on Synthetic Data? A Statistical Analysis of Language Model Collapse](https://mlsanity.com/paper/2404.05090/) (2024)
- [Fairness Feedback Loops: Training on Synthetic Data Amplifies Bias](https://mlsanity.com/paper/2403.07857/) (2024)
- [Collapse or Thrive? Perils and Promises of Synthetic Data in a Self-Generating World](https://mlsanity.com/paper/2410.16713/) (2024)
- [Combining Generative Artificial Intelligence (AI) and the Internet: Heading towards Evolution or Degradation?](https://mlsanity.com/paper/2303.01255/) (2023)
- [Self-Correcting Self-Consuming Loops for Generative Model Training](https://mlsanity.com/paper/2402.07087/) (2024)
- [Position: Model Collapse Does Not Mean What You Think](https://mlsanity.com/paper/2503.03150/) (2025)
- [Nepotistically Trained Generative-AI Models Collapse](https://mlsanity.com/paper/2311.12202/) (2023)
- [Rate of Model Collapse in Recursive Training](https://mlsanity.com/paper/2412.17646/) (2024)
- [Tracking the perspectives of interacting language models](https://mlsanity.com/paper/2406.11938/) (2024)
- [Recursive Learning Without Collapse: A Weighting-Based Stabilization Framework](https://mlsanity.com/paper/2502.18049/) (2025)
- [Multilingual Performance Biases of Large Language Models in Education](https://mlsanity.com/paper/2504.17720/) (2025)

## Built on by (most cited first, 6 of 6)

- [Unsupervised Conformal Inference: Bootstrapping and Alignment to Control LLM Uncertainty](https://mlsanity.com/paper/2509.23002/) (2025)
- [Epistemic diversity across language models mitigates knowledge collapse](https://mlsanity.com/paper/2512.15011/) (2025)
- [The Anti-Ouroboros Effect: Emergent Resilience in Large Language Models from Recursive Selective Feedback](https://mlsanity.com/paper/2509.10509/) (2025)
- [SIGMA: Scalable Spectral Insights for LLM Model Collapse](https://mlsanity.com/paper/2601.03385/) (2026)
- [RAG Collapse: LLM Responses Collapse When Retrieved Documents Are Self-Authored](https://mlsanity.com/paper/2608.22118/) (2026)
- [The Oligarch Barely Steers Model Collapse in Multi-Model Ecosystems](https://mlsanity.com/paper/2609.11146/) (2026)

## Abstract (arXiv)

The increasing use of synthetic data from the public Internet has enhanced data usage efficiency in large language model (LLM) training. However, the potential threat of model collapse remains insufficiently explored. Existing studies primarily examine model collapse in a single model setting or rely solely on statistical surrogates. In this work, we introduce LLM Web Dynamics (LWD), an efficient framework for investigating model collapse at the network level. By simulating the Internet with a retrieval-augmented generation (RAG) database, we analyze the convergence pattern of model outputs. Furthermore, we provide theoretical guarantees for this convergence by drawing an analogy to interacting Gaussian Mixture Models.