The Oligarch Barely Steers Model Collapse in Multi-Model Ecosystems
On the map: Synthetic data and self-improvement for LLMs → Model collapse from synthetic data
Abstract (arXiv)
More and more of the text future language models learn from is written by a few of today's models. If one supplier writes most of a shared corpus, does it pull the models trained on it toward its own writing, or change how fast they drift? We retrain eight open models from their base weights on a shared pool of each other's text for five generations, varying the part written by one model, Phi-2, from an equal share to 90%. The models drift together toward a style with fewer function words, and none starts repeating itself. No share of Phi-2 brings the other models closer to its text than the equal share does. We split each ecosystem's separation from the equal-share one into a delay along its route and a departure from that route, both counted beyond the difference between two equal-share runs. With Phi-2 at 90%, delay outweighs departure 72 to 28 and 64 to 36 in two runs, and the ecosystem falls 2.7 and 2.5 generations behind. With Phi-2 at half the pool the two parts are about equal. When SmolLM2 or Qwen3-1.7B writes half instead, the ecosystem slows less or not at all. The departure leans toward Phi-2 more as its share grows, but more than toward every other model only at 90%. Human text filling a quarter or half of the pool slows the models along the same route.
Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in Model collapse from synthetic data, one of 13 directions in Synthetic data and self-improvement for LLMs. Builds on 24 of its 38 references that are in the map. First posted on arXiv 10 Sep 2026.
Builds on
24 of 38 references.
Closest ancestors · 7
Same direction, newest first
- 2026When Does Model Collapse Occur in Structured Interactive Learning?19 May
- 2025LLM Web Dynamics: Tracing Model Collapse in a Network of LLMs26 May
- 2025Position: Model Collapse Does Not Mean What You Think5 Mar
- 2024Beyond Model Collapse: Scaling Up with Synthesized Data Requires Verification11 Jun
- 2024How Bad is Training on Synthetic Data? A Statistical Analysis of Language Model Collapse7 Apr
- 2024Is Model Collapse Inevitable? Breaking the Curse of Recursion by Accumulating Real and Synthetic Data1 Apr
- 2024A Tale of Tails: Model Collapse as a Change of Scaling Laws10 Feb
Other references · 17
- 2023Efficient Memory Management for Large Language Model Serving with PagedAttentioncited by 5,944
- 2024Qwen2.5 Technical Reportcited by 4,942
- 2019The Curious Case of Neural Text Degenerationcited by 2,916
- 2020The Pile: An 800GB Dataset of Diverse Text for Language Modelingcited by 2,411
- 2021DeBERTaV3: Improving DeBERTa using ELECTRA-Style Pre-Training with Gradient-Disentangled Embedding Sharingcited by 1,254
- 2015A Diversity-Promoting Objective Function for Neural Conversation Modelscited by 1,077
- 2025SmolLM2: When Smol Goes Big -- Data-Centric Training of a Small Language Modelcited by 335
- 2023Self-Consuming Generative Models Go MADcited by 250
- 2021Algorithmic Monoculture and Social Welfarecited by 100
- 2023On the Stability of Iterative Retraining of Generative Models on their own Datacited by 100
- 2023The Curious Decline of Linguistic Diversity: Training Language Models on Synthetic Textcited by 100
- 2022Picking on the Same Person: Does Algorithmic Monoculture lead to Outcome Homogenization?cited by 91
5 more
- 2024Collapse or Thrive? Perils and Promises of Synthetic Data in a Self-Generating Worldcited by 52
- 2024Self-Correcting Self-Consuming Loops for Generative Model Trainingcited by 37
- 2024Generative Monoculture in Large Language Modelscited by 22
- 2025What happens when generative AI models train recursively on each others' outputs?cited by 6
- 2025Epistemic diversity across language models mitigates knowledge collapsecited by 2
Built on by
Not cited yet.
Similar, not cited
Close in wording, no citation either way
- 2026Position: the Stochastic Parrot in the Coal Mine. Model Collapse is a Threat to Low-Resource Communities
- 2026Reviewing Model Collapse and Countermeasures
- 2026The Institutional Scaling Law: Non-Monotonic Fitness, Capability-Trust Divergence, and Symbiogenetic Scaling in Generative AI
- 2024A Note on Shumailov et al. (2024): `AI Models Collapse When Trained on Recursively Generated Data'
- 2026The evolution of sex for artificial intelligence: a population-genetic framework for multigenerational model populations
- 2026The Headless Firm: How AI Reshapes Enterprise Boundaries
- 2026Increasing intelligence in AI agents can worsen collective outcomes
- 2026Model Collapse as Cultural Evolution