# Universality of the π2/6 Pathway in Avoiding Model Collapse

arXiv: https://arxiv.org/abs/2410.22812 · HTML: https://mlsanity.com/paper/2410.22812/

## Key facts

- Cited by 26 papers in the map (data through 14 Oct 2026).
- Builds on 21 of its 28 references that are in the map.
- First posted on arXiv 30 Oct 2024.

## Builds on (most cited first, 21 of 21)

- [A Simple Framework for Contrastive Learning of Visual Representations](https://mlsanity.com/paper/2002.05709/) (2020)
- [Momentum Contrast for Unsupervised Visual Representation Learning](https://mlsanity.com/paper/1911.05722/) (2019)
- [Emerging Properties in Self-Supervised Vision Transformers](https://mlsanity.com/paper/2104.14294/) (2021)
- [Unsupervised Learning of Visual Features by Contrasting Cluster Assignments](https://mlsanity.com/paper/2006.09882/) (2020)
- [The Curse of Recursion: Training on Generated Data Makes Models Forget](https://mlsanity.com/paper/2305.17493/) (2023)
- [Self-Consuming Generative Models Go MAD](https://mlsanity.com/paper/2307.01850/) (2023)
- [Self-Distillation Amplifies Regularization in Hilbert Space](https://mlsanity.com/paper/2002.05715/) (2020)
- [Is Model Collapse Inevitable? Breaking the Curse of Recursion by Accumulating Real and Synthetic Data](https://mlsanity.com/paper/2404.01413/) (2024)
- [A Tale of Tails: Model Collapse as a Change of Scaling Laws](https://mlsanity.com/paper/2402.07043/) (2024)
- [On the Stability of Iterative Retraining of Generative Models on their own Data](https://mlsanity.com/paper/2310.00429/) (2023)
- [Model Collapse Demystified: The Case of Regression](https://mlsanity.com/paper/2402.07712/) (2024)
- [How Bad is Training on Synthetic Data? A Statistical Analysis of Language Model Collapse](https://mlsanity.com/paper/2404.05090/) (2024)
- [Will Large-scale Generative Models Corrupt Future Datasets?](https://mlsanity.com/paper/2211.08095/) (2022)
- [Towards Understanding the Interplay of Generative Artificial Intelligence and the Internet](https://mlsanity.com/paper/2306.06130/) (2023)
- [Collapse or Thrive? Perils and Promises of Synthetic Data in a Self-Generating World](https://mlsanity.com/paper/2410.16713/) (2024)
- [Combining Generative Artificial Intelligence (AI) and the Internet: Heading towards Evolution or Degradation?](https://mlsanity.com/paper/2303.01255/) (2023)
- [Self-Consuming Generative Models with Curated Data Provably Optimize Human Preferences](https://mlsanity.com/paper/2407.09499/) (2024)
- [Self-Improving Diffusion Models with Synthetic Data](https://mlsanity.com/paper/2408.16333/) (2024)
- [Scaling laws for learning with real and surrogate data](https://mlsanity.com/paper/2402.04376/) (2024)
- [Heat Death of Generative Models in Closed-Loop Learning](https://mlsanity.com/paper/2404.02325/) (2024)
- [One Step to Efficient Synthetic Data](https://mlsanity.com/paper/2006.02397/) (2020)

## Built on by (most cited first, 26 of 26)

- [Position: Model Collapse Does Not Mean What You Think](https://mlsanity.com/paper/2503.03150/) (2025)
- [A Closer Look at Model Collapse: From a Generalization-to-Memorization Perspective](https://mlsanity.com/paper/2509.16499/) (2025)
- [When Models Don't Collapse: On the Consistency of Iterative MLE](https://mlsanity.com/paper/2505.19046/) (2025)
- [Escaping Collapse: The Strength of Weak Data for Large Language Model Training](https://mlsanity.com/paper/2502.08924/) (2025)
- [Recursive Learning Without Collapse: A Weighting-Based Stabilization Framework](https://mlsanity.com/paper/2502.18049/) (2025)
- [Preventing Model Collapse Under Overparametrization: Optimal Mixing Ratios for Interpolation Learning and Ridge Regression](https://mlsanity.com/paper/2509.22341/) (2025)
- [Position: Machine Learning Conferences Should Establish a "Refutations and Critiques" Track](https://mlsanity.com/paper/2506.19882/) (2025)
- [What happens when generative AI models train recursively on each others' outputs?](https://mlsanity.com/paper/2505.21677/) (2025)
- [Escaping Model Collapse via Synthetic Data Verification: Near-term Improvements and Long-term Convergence](https://mlsanity.com/paper/2510.16657/) (2025)
- [Do Large Language Models (Really) Need Statistical Foundations?](https://mlsanity.com/paper/2505.19145/) (2025)
- [Language Generation with Replay: A Learning-Theoretic View of Model Collapse](https://mlsanity.com/paper/2603.11784/) (2026)
- [A Probabilistic Perspective on Model Collapse](https://mlsanity.com/paper/2505.13947/) (2025)
- [Quantifying Error Propagation and Model Collapse in Diffusion Models](https://mlsanity.com/paper/2602.16601/) (2026)
- [On the Generalization Gap in Self-Evolving Language Model Reasoning](https://mlsanity.com/paper/2606.01075/) (2026)
- [Learning from Synthetic Data: Limitations of ERM](https://mlsanity.com/paper/2601.15468/) (2026)
- [When Does Model Collapse Occur in Structured Interactive Learning?](https://mlsanity.com/paper/2605.20151/) (2026)
- [We Need Improved Data Curation and Attribution in AI for Scientific Discovery](https://mlsanity.com/paper/2504.02486/) (2025)
- [What Matters in LLM-generated Data: Diversity and Its Effect on Model Fine-Tuning](https://mlsanity.com/paper/2506.19262/) (2025)
- [Why Self-Training Helps and Hurts: Denoising vs. Signal Forgetting](https://mlsanity.com/paper/2602.14029/) (2026)
- [When Sample Selection Bias Precipitates Model Collapse](https://mlsanity.com/paper/2606.13732/) (2026)
- [Maximum Likelihood Estimation for Entity Ranking under Iterative Synthetic Data Augmentation](https://mlsanity.com/paper/2609.32987/) (2026)
- [Can Generative Artificial Intelligence Survive Data Contamination? Theoretical Guarantees under Contaminated Recursive Training](https://mlsanity.com/paper/2602.16065/) (2026)
- [Experimental evidence of progressive ChatGPT models self-convergence](https://mlsanity.com/paper/2603.12683/) (2026)
- [Recursively Trained Diffusion Models: Limiting Collapse Distribution and Spectral Characterization](https://mlsanity.com/paper/2606.13796/) (2026)
- [Pushing the Limits of High-Resolution Weather Forecasting through Data Scaling](https://mlsanity.com/paper/2608.14652/) (2026)
- [Self-Cleaning and Captured Anyway: One Measured Primitive for Error in a Store an Agent Writes to Itself, and What a Falling Score Actually Measures](https://mlsanity.com/paper/2609.25052/) (2026)

## Abstract (arXiv)

Researchers in empirical machine learning recently spotlighted their fears of so-called Model Collapse. They imagined a discard workflow, where an initial generative model is trained with real data, after which the real data are discarded, and subsequently, the model generates synthetic data on which a new model is trained. They came to the conclusion that models degenerate as model-fitting generations proceed. However, other researchers considered an augment workflow, where the original real data continue to be used in each generation of training, augmented by synthetic data from models fit in all earlier generations. Empirical results on canonical datasets and learning procedures confirmed the occurrence of model collapse under the discard workflow and avoidance of model collapse under the augment workflow. Under the augment workflow, theoretical evidence also confirmed avoidance in particular instances; specifically, Gerstgrasser et al. (2024) found that for classical Linear Regression, test risk at any later generation is bounded by a moderate multiple, viz. pi-squared-over-6 of the test risk of training with the original real data alone. Some commentators questioned the generality of theoretical conclusions based on the generative model assumed in Gerstgrasser et al. (2024): could similar conclusions be reached for other task/model pairings? In this work, we demonstrate the universality of the pi-squared-over-6 augment risk bound across a large family of canonical statistical models, offering key insights into exactly why collapse happens under the discard workflow and is avoided under the augment workflow. In the process, we provide a framework that is able to accommodate a large variety of workflows (beyond discard and augment), thereby enabling an experimenter to judge the comparative merits of multiple different workflows by simulating a simple Gaussian process.