mlsanityData to 7 Oct 2026

Convergence and Stability Analysis of Self-Consuming Generative Models with Heterogeneous Human Curation

arXiv 2511.09002 ↗First posted 12 Nov 2025stat.ML cs.LGcited by 2

Not on the map. Why?

Abstract (arXiv)

Self-consuming generative models have received significant attention over the last few years. In this paper, we study a self-consuming generative model with heterogeneous preferences that is a generalization of the model in Ferbach et al. (2024). The model is retrained round by round using real data and its previous-round synthetic outputs. The asymptotic behavior of the retraining dynamics is investigated across four regimes using different techniques including the nonlinear Perron--Frobenius theory. Our analyses improve upon that of Ferbach et al. (2024) and provide convergence results in settings where the well-known Banach contraction mapping arguments do not apply. Stability and non-stability results regarding the retraining dynamics are also given.

Key facts. Cited by 2 papers in the map (data through 7 Oct 2026). Builds on 42 of its 60 references that are in the map. First posted on arXiv 12 Nov 2025.

Builds on

42 of 60 references.

References · 42

  1. 2022Training language models to follow instructions with human feedbackcited by 15,712
  2. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  3. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 7,916
  4. 2022LAION-5B: An open large-scale dataset for training next generation image-text modelscited by 4,131
  5. 2017Deep reinforcement learning from human preferencescited by 3,846
  6. 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedbackcited by 3,634
  7. 2022DreamBooth: Fine Tuning Text-to-Image Diffusion Models for Subject-Driven Generationcited by 3,286
  8. 2022Self-Instruct: Aligning Language Models with Self-Generated Instructionscited by 2,674
  9. 2020Learning to summarize from human feedbackcited by 2,562
  10. 2021WebGPT: Browser-assisted question-answering with human feedbackcited by 1,722
  11. 2022Red Teaming Language Models with Language Modelscited by 983
  12. 2024SimPO: Simple Preference Optimization with a Reference-Free Rewardcited by 951
30 more
  1. 2023A General Theoretical Paradigm to Understand Learning from Human Preferencescited by 906
  2. 2023OpenAssistant Conversations -- Democratizing Large Language Model Alignmentcited by 712
  3. 2024ORPO: Monolithic Preference Optimization without Reference Modelcited by 556
  4. 2022Safe Latent Diffusion: Mitigating Inappropriate Degeneration in Diffusion Modelscited by 501
  5. 2023Weak-to-Strong Generalization: Eliciting Strong Capabilities With Weak Supervisioncited by 445
  6. 2023Understanding the Effects of RLHF on LLM Generalisation and Diversitycited by 386
  7. 2021Multimodal datasets: misogyny, pornography, and malignant stereotypescited by 326
  8. 2021PEBBLE: Feedback-Efficient Interactive Reinforcement Learning via Relabeling Experience and Unsupervised Pre-trainingcited by 298
  9. 2023A Survey of Reinforcement Learning from Human Feedbackcited by 259
  10. 2024KTO: Model Alignment as Prospect Theoretic Optimizationcited by 256
  11. 2024Iterative Reasoning Preference Optimizationcited by 244
  12. 2023Large Language Model Alignment: A Surveycited by 241
  13. 2023Nash Learning from Human Feedbackcited by 220
  14. 2016Professor Forcing: A New Algorithm for Training Recurrent Networkscited by 214
  15. 2023Better Diffusion Models Further Improve Adversarial Trainingcited by 207
  16. 2016Dual Learning for Machine Translationcited by 146
  17. 2019Learning from Dialogue after Deployment: Feed Yourself, Chatbot!cited by 102
  18. 2023On the Stability of Iterative Retraining of Generative Models on their own Datacited by 98
  19. 2024RLHF from Heterogeneous Feedback via Personalization and Preference Aggregationcited by 71
  20. 2022Will Large-scale Generative Models Corrupt Future Datasets?cited by 59
  21. 2023Benchmarks and Algorithms for Offline Preference-Based Reward Learningcited by 55
  22. 2023Into the LAIONs Den: Investigating Hate in Multimodal Datasetscited by 54
  23. 2023Combining Generative Artificial Intelligence (AI) and the Internet: Heading towards Evolution or Degradation?cited by 41
  24. 2024Online DPO: Online Direct Preference Optimization with Fast-Slow Chasingcited by 38
  25. 2024Self-Consuming Generative Models with Curated Data Provably Optimize Human Preferencescited by 35
  26. 2024Diffusion Models as Constrained Samplers for Optimization with Unknown Constraintscited by 33
  27. 2023Nepotistically Trained Generative-AI Models Collapsecited by 27
  28. 2023A density estimation perspective on learning from pairwise human preferencescited by 23
  29. 2024The Road to Artificial SuperIntelligence: A Comprehensive Survey of Superalignmentcited by 11
  30. 2022Deep Generative Modeling on Limited Data with Regularization by Nontransferable Pre-trained Modelscited by 4

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2026: 2.

2 papers.

  1. 2026Curated Synthetic Data Doesn't Have to Collapse: A Theoretical Study of Generative Retraining with Pluralistic Preferencescited by 0
  2. 2026When and How Human Curation Backfires: Preference Alignment under Multi-Model Self-Consuming Loopcited by 0