mlsanityData to 14 Oct 2026

LLM Web Dynamics: Tracing Model Collapse in a Network of LLMs

arXiv 2506.15690 ↗First posted 26 May 2025cs.LG cs.AI cs.SIcited by 6

On the map: Synthetic data and self-improvement for LLMs → Model collapse from synthetic data

Abstract (arXiv)

The increasing use of synthetic data from the public Internet has enhanced data usage efficiency in large language model (LLM) training. However, the potential threat of model collapse remains insufficiently explored. Existing studies primarily examine model collapse in a single model setting or rely solely on statistical surrogates. In this work, we introduce LLM Web Dynamics (LWD), an efficient framework for investigating model collapse at the network level. By simulating the Internet with a retrieval-augmented generation (RAG) database, we analyze the convergence pattern of model outputs. Furthermore, we provide theoretical guarantees for this convergence by drawing an analogy to interacting Gaussian Mixture Models.

Key facts. Cited by 6 papers in the map (data through 14 Oct 2026). Placed in Model collapse from synthetic data, one of 13 directions in Synthetic data and self-improvement for LLMs. Builds on 24 of its 46 references that are in the map. First posted on arXiv 26 May 2025.

Builds on

24 of 46 references.

Closest ancestors · 5

Same direction, newest first

  1. 2025Position: Model Collapse Does Not Mean What You Think5 Mar
  2. 2025Recursive Learning Without Collapse: A Weighting-Based Stabilization Framework25 Feb
  3. 2024Rate of Model Collapse in Recursive Training23 Dec
  4. 2024How Bad is Training on Synthetic Data? A Statistical Analysis of Language Model Collapse7 Apr
  5. 2023The Curse of Recursion: Training on Generated Data Makes Models Forget27 May

Other references · 19

  1. 2020Language Models are Few-Shot Learnerscited by 32,419
  2. 2020Scaling Laws for Neural Language Modelscited by 6,586
  3. 2023Retrieval-Augmented Generation for Large Language Models: A Surveycited by 2,212
  4. 2023OpenChat: Advancing Open-source Language Models with Mixed-Quality Datacited by 252
  5. 2023Self-Consuming Generative Models Go MADcited by 250
  6. 2025Agentic Retrieval-Augmented Generation: A Survey on Agentic RAGcited by 249
  7. 2022Will we run out of data? Limits of LLM scaling based on human-generated datacited by 195
  8. 2020Tensor Programs II: Neural Tangent Kernel for Any Architecturecited by 143
  9. 2019On the Turing Completeness of Modern Neural Network Architecturescited by 129
  10. 2024A Note on Shumailov et al. (2024): `AI Models Collapse When Trained on Recursively Generated Data'cited by 115
  11. 2023On the Stability of Iterative Retraining of Generative Models on their own Datacited by 100
  12. 2024RAG vs Fine-tuning: Pipelines, Tradeoffs, and a Case Study on Agriculturecited by 81
7 more
  1. 2024Fairness Feedback Loops: Training on Synthetic Data Amplifies Biascited by 55
  2. 2024Collapse or Thrive? Perils and Promises of Synthetic Data in a Self-Generating Worldcited by 52
  3. 2023Combining Generative Artificial Intelligence (AI) and the Internet: Heading towards Evolution or Degradation?cited by 41
  4. 2024Self-Correcting Self-Consuming Loops for Generative Model Trainingcited by 37
  5. 2023Nepotistically Trained Generative-AI Models Collapsecited by 27
  6. 2024Tracking the perspectives of interacting language modelscited by 14
  7. 2025Multilingual Performance Biases of Large Language Models in Educationcited by 9

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 3; 2026: 3.

6 papers.

  1. 2025Unsupervised Conformal Inference: Bootstrapping and Alignment to Control LLM Uncertaintycited by 2
  2. 2025Epistemic diversity across language models mitigates knowledge collapsecited by 2
  3. 2025The Anti-Ouroboros Effect: Emergent Resilience in Large Language Models from Recursive Selective Feedbackcited by 0
  4. 2026SIGMA: Scalable Spectral Insights for LLM Model Collapsecited by 0
  5. 2026RAG Collapse: LLM Responses Collapse When Retrieved Documents Are Self-Authoredcited by 0
  6. 2026The Oligarch Barely Steers Model Collapse in Multi-Model Ecosystemscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026Beyond Pipelines: A Fundamental Study on the Rise of Generative-Retrieval Architectures in Web Research
  2. 2024Model-GLUE: Democratized LLM Scaling for A Large Model Zoo in the Wild
  3. 2025LLMs on the Line: Data Determines Loss-to-Loss Scaling Laws
  4. 2025Demystifying Synthetic Data in LLM Pre-training: A Systematic Study of Scaling Laws, Benefits, and Pitfalls
  5. 2025What Matters in LLM-generated Data: Diversity and Its Effect on Model Fine-Tuning
  6. 2026Language Generation with Replay: A Learning-Theoretic View of Model Collapse
  7. 2026Infinite-Parameter LLMs: Generating and Adapting Weights from Live Data
  8. 2023To Repeat or Not To Repeat: Insights from Scaling LLM under Token-Crisis