mlsanityData to 7 Oct 2026

Self-Correcting Self-Consuming Loops for Generative Model Training

arXiv 2402.07087 ↗First posted 11 Feb 2024cs.LG cs.AI cs.CVcited by 34

Not on the map. Why?

Abstract (arXiv)

As synthetic data becomes higher quality and proliferates on the internet, machine learning models are increasingly trained on a mix of human- and machine-generated data. Despite the successful stories of using synthetic data for representation learning, using synthetic data for generative model training creates "self-consuming loops" which may lead to training instability or even collapse, unless certain conditions are met. Our paper aims to stabilize self-consuming generative model training. Our theoretical results demonstrate that by introducing an idealized correction function, which maps a data point to be more likely under the true data distribution, self-consuming loops can be made exponentially more stable. We then propose self-correction functions, which rely on expert knowledge (e.g. the laws of physics programmed in a simulator), and aim to approximate the idealized corrector automatically and at scale. We empirically validate the effectiveness of self-correcting self-consuming loops on the challenging human motion synthesis task, and observe that it successfully avoids model collapse, even when the ratio of synthetic data to real data is as high as 100%.

Key facts. Cited by 34 papers in the map (data through 7 Oct 2026). Builds on 24 of its 34 references that are in the map. First posted on arXiv 11 Feb 2024.

Builds on

24 of 34 references.

References · 24

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  2. 2020Denoising Diffusion Probabilistic Modelscited by 20,851
  3. 2022Photorealistic Text-to-Image Diffusion Models with Deep Language Understandingcited by 6,322
  4. 2022Classifier-Free Diffusion Guidancecited by 5,717
  5. 2022LAION-5B: An open large-scale dataset for training next generation image-text modelscited by 4,131
  6. 2019Expressive Body Capture: 3D Hands, Face, and Body from a Single Imagecited by 1,600
  7. 2022Human Motion Diffusion Modelcited by 1,062
  8. 2022Self-critiquing models for assisting human evaluatorscited by 372
  9. 2022PhysDiff: Physics-Guided Human Motion Diffusion Modelcited by 365
  10. 2023The Curse of Recursion: Training on Generated Data Makes Models Forgetcited by 343
  11. 2023Synthetic Data from Diffusion Models Improves ImageNet Classificationcited by 326
  12. 2022Is synthetic data from generative models ready for image recognition?cited by 320
12 more
  1. 2020Performative Predictioncited by 318
  2. 2022Generating Sequences by Learning to Self-Correctcited by 271
  3. 2023Self-Consuming Generative Models Go MADcited by 239
  4. 2023InterDiff: Generating 3D Human-Object Interactions with Physics-Informed Diffusioncited by 181
  5. 2023Self-correcting LLM-controlled Diffusion Modelscited by 112
  6. 2021Generative Models as a Data Source for Multiview Representation Learningcited by 104
  7. 2021Dynamics-Regulated Kinematic Policy for Egocentric Pose Estimationcited by 101
  8. 2023On the Stability of Iterative Retraining of Generative Models on their own Datacited by 98
  9. 2023Learning Vision from Models Rivals Learning Vision from Datacited by 71
  10. 2020MoVi: A Large Multipurpose Motion and Video Datasetcited by 58
  11. 2023Towards Understanding the Interplay of Generative Artificial Intelligence and the Internetcited by 54
  12. 2018Generalization Error in Deep Learningcited by 46

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 9; 2025: 12; 2026: 13.

34 papers.

Sort:

By area:

  1. 2024Model Collapse Demystified: The Case of Regressioncited by 80
  2. 2025Can Large Reasoning Models Self-Train?cited by 66
  3. 2024Strong Model Collapsecited by 62
  4. 2024Collapse or Thrive? Perils and Promises of Synthetic Data in a Self-Generating Worldcited by 49
  5. 2024Beyond Model Collapse: Scaling Up with Synthesized Data Requires Verificationcited by 43
  6. 2024Self-Consuming Generative Models with Curated Data Provably Optimize Human Preferencescited by 35
  7. 2024Self-Improving Diffusion Models with Synthetic Datacited by 34
  8. 2025Position: Model Collapse Does Not Mean What You Thinkcited by 32
  9. 2025Self-Improving Transformers Overcome Easy-to-Hard and Length Generalization Challengescited by 30
  10. 2024Mind the Gap: Examining the Self-Improvement Capabilities of Large Language Modelscited by 16
  11. 2025A Theoretical Perspective: How to Prevent Model Collapse in Self-consuming Training Loopscited by 15
  12. 2024Model Collapse in the Self-Consuming Chain of Diffusion Finetuning: A Novel Perspective from Quantitative Trait Modelingcited by 11
22 more
  1. 2025Escaping Collapse: The Strength of Weak Data for Large Language Model Trainingcited by 11
  2. 2025Self-Consuming Generative Models with Adversarially Curated Datacited by 8
  3. 2025Why Less is More (Sometimes): A Theory of Data Curationcited by 8
  4. 2025ModSkill: Physical Character Skill Modularizationcited by 7
  5. 2025LLM Web Dynamics: Tracing Model Collapse in a Network of LLMscited by 6
  6. 2024ACDC: Autoregressive Coherent Multimodal Generation using Diffusion Correctioncited by 3
  7. 2025Spend Wisely: Maximizing Post-Training Gains in Iterative Synthetic Data Bootstrappingcited by 2
  8. 2026Ambient Dataloops: Generative Models for Dataset Refinementcited by 2
  9. 2026When AI Reviews Its Own Code: Recursive Self-Training Collapse in Code LLMscited by 2
  10. 2025Convergence Dynamics and Stabilization Strategies of Co-Evolving Generative Modelscited by 1
  11. 2025SimDiff: Simulator-constrained Diffusion Model for Physically Plausible Motion Generationcited by 1
  12. 2026Observations and Remedies for Large Language Model Bias in Self-Consuming Performative Loopcited by 1
  13. 2026Large Language Models are Shannon Lossy Compressors Not Solomonoff Induction Estimators: Self-improvement and Singularity Are Not Near Without Symbolic Model Synthesiscited by 1
  14. 2026Drift and selection in LLM text ecosystemscited by 1
  15. 2026Beyond MoCap: Scaling Motion Tokenizers with Synthetic Human Motion for Generative Modelingcited by 1
  16. 2026Skeleton2Stage: Reward-Guided Fine-Tuning for Physically Plausible Dance Generationcited by 0
  17. 2026A Markovian View of Iterative-Feedback Loops in Image Generative Models: Neural Resonance and Model Collapsecited by 0
  18. 2026The Oligarch Barely Steers Model Collapse in Multi-Model Ecosystemscited by 0
  19. 2026Break Step: Recursive Training Resonates with Replayed Sampling Noisecited by 0
  20. 2026SynthSentry: Detecting Synthetic Data Contamination in Language Model Training Datacited by 0
  21. 2026Recursive LLM Degradation in Biomedical Question Answering: A Cross-Generation Studycited by 0
  22. 2026Source Identification Is Not Fitness Testing: Measuring the Limits of Synthetic-Data Attributioncited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Escaping Model Collapse via Synthetic Data Verification: Near-term Improvements and Long-term Convergence
  2. 2025Recursive Learning Without Collapse: A Weighting-Based Stabilization Framework
  3. 2024Towards Theoretical Understandings of Self-Consuming Generative Models
  4. 2025Stabilizing Self-Consuming Diffusion Models with Latent Space Filtering
  5. 2026Preventing Model Collapse: A Fisher-Rao Perspective on the Dynamics of Training with Synthetic Data
  6. 2024Analyzing and Mitigating Model Collapse in Rectified Flow Models
  7. 2025Fine-Tuning Masked Diffusion for Provable Self-Correction
  8. 2023Real-Fake: Effective Training Data Synthesis Through Distribution Matching