# Recursive Learning Without Collapse: A Weighting-Based Stabilization Framework

arXiv: https://arxiv.org/abs/2502.18049 · HTML: https://mlsanity.com/paper/2502.18049/

## Key facts

- Cited by 10 papers in the map (data through 14 Oct 2026).
- Placed in Model collapse from synthetic data, one of 13 directions in Synthetic data and self-improvement for LLMs.
- Builds on 19 of its 32 references that are in the map.
- First posted on arXiv 25 Feb 2025.

## Placed in

- Synthetic data and self-improvement for LLMs / [Model collapse from synthetic data](https://mlsanity.com/synth/model-collapse-from-synthetic-data/)

## Builds on (most cited first, 19 of 19)

- [GPT-4 Technical Report](https://mlsanity.com/paper/2303.08774/) (2023)
- [DetectGPT: Zero-Shot Machine-Generated Text Detection using Probability Curvature](https://mlsanity.com/paper/2301.11305/) (2023)
- [The Curse of Recursion: Training on Generated Data Makes Models Forget](https://mlsanity.com/paper/2305.17493/) (2023)
- [Self-Consuming Generative Models Go MAD](https://mlsanity.com/paper/2307.01850/) (2023)
- [Generating Training Data with Language Models: Towards Zero-Shot Language Understanding](https://mlsanity.com/paper/2202.04538/) (2022)
- [Is Model Collapse Inevitable? Breaking the Curse of Recursion by Accumulating Real and Synthetic Data](https://mlsanity.com/paper/2404.01413/) (2024)
- [A Tale of Tails: Model Collapse as a Change of Scaling Laws](https://mlsanity.com/paper/2402.07043/) (2024)
- [On the Stability of Iterative Retraining of Generative Models on their own Data](https://mlsanity.com/paper/2310.00429/) (2023)
- [Model Collapse Demystified: The Case of Regression](https://mlsanity.com/paper/2402.07712/) (2024)
- [Understanding data augmentation for classification: when to warp?](https://mlsanity.com/paper/1609.08764/) (2016)
- [Strong Model Collapse](https://mlsanity.com/paper/2410.04840/) (2024)
- [A Statistical Framework of Watermarks for Large Language Models: Pivot, Detection Efficiency and Optimal Rules](https://mlsanity.com/paper/2404.01245/) (2024)
- [Collapse or Thrive? Perils and Promises of Synthetic Data in a Self-Generating World](https://mlsanity.com/paper/2410.16713/) (2024)
- [Robust Detection of Watermarks for Large Language Models Under Human Edits](https://mlsanity.com/paper/2411.13868/) (2024)
- [Data Augmentation using Large Language Models: Data Perspectives, Learning Paradigms and Challenges](https://mlsanity.com/paper/2403.02990/) (2024)
- [Universality of the π2/6 Pathway in Avoiding Model Collapse](https://mlsanity.com/paper/2410.22812/) (2024)
- [Debiasing Watermarks for Large Language Models via Maximal Coupling](https://mlsanity.com/paper/2411.11203/) (2024)
- [Towards Better Statistical Understanding of Watermarking LLMs](https://mlsanity.com/paper/2403.13027/) (2024)
- [Utility Theory of Synthetic Data Generation](https://mlsanity.com/paper/2305.10015/) (2023)

## Built on by (most cited first, 10 of 10)

- [When Models Don't Collapse: On the Consistency of Iterative MLE](https://mlsanity.com/paper/2505.19046/) (2025)
- [Preventing Model Collapse Under Overparametrization: Optimal Mixing Ratios for Interpolation Learning and Ridge Regression](https://mlsanity.com/paper/2509.22341/) (2025)
- [LLM Web Dynamics: Tracing Model Collapse in a Network of LLMs](https://mlsanity.com/paper/2506.15690/) (2025)
- [Escaping Model Collapse via Synthetic Data Verification: Near-term Improvements and Long-term Convergence](https://mlsanity.com/paper/2510.16657/) (2025)
- [A Probabilistic Perspective on Model Collapse](https://mlsanity.com/paper/2505.13947/) (2025)
- [When Does Model Collapse Occur in Structured Interactive Learning?](https://mlsanity.com/paper/2605.20151/) (2026)
- [Why Self-Training Helps and Hurts: Denoising vs. Signal Forgetting](https://mlsanity.com/paper/2602.14029/) (2026)
- [Maximum Likelihood Estimation for Entity Ranking under Iterative Synthetic Data Augmentation](https://mlsanity.com/paper/2609.32987/) (2026)
- [Experimental evidence of progressive ChatGPT models self-convergence](https://mlsanity.com/paper/2603.12683/) (2026)
- [Prediction-Only Distillation in Linear and Logistic Regression](https://mlsanity.com/paper/2607.15450/) (2026)

## Abstract (arXiv)

Recent studies identified an intriguing phenomenon in recursive generative model training known as model collapse, where models trained on data generated by previous models exhibit severe performance degradation. Addressing this issue and developing more effective training strategies have become central challenges in generative model research. In this paper, we investigate this phenomenon within a novel framework, where generative models are iteratively trained on a combination of newly collected real data and synthetic data from the previous training step. To develop an optimal training strategy for integrating real and synthetic data, we evaluate the performance of a weighted training scheme in various scenarios, including Gaussian distribution estimation, generalized linear models, and nonparametric estimation. We theoretically characterize the impact of the mixing proportion and weighting scheme of synthetic data on the final model's performance. Our key finding is that, across different settings, the optimal weighting scheme under different proportions of synthetic data asymptotically follows a unified expression, revealing a fundamental trade-off between leveraging synthetic data and model performance. In some cases, the optimal weight assigned to real data corresponds to the reciprocal of the golden ratio. Finally, we validate our theoretical results on extensive simulated datasets and a real tabular dataset.