# When and How Human Curation Backfires: Preference Alignment under Multi-Model Self-Consuming Loop

arXiv: https://arxiv.org/abs/2605.29267 · HTML: https://mlsanity.com/paper/2605.29267/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 51 of its 64 references that are in the map.
- First posted on arXiv 28 May 2026.

## Builds on (most cited first, 30 of 51)

- [Denoising Diffusion Probabilistic Models](https://mlsanity.com/paper/2006.11239/) (2020)
- [Training language models to follow instructions with human feedback](https://mlsanity.com/paper/2203.02155/) (2022)
- [LoRA: Low-Rank Adaptation of Large Language Models](https://mlsanity.com/paper/2106.09685/) (2021)
- [Denoising Diffusion Implicit Models](https://mlsanity.com/paper/2010.02502/) (2020)
- [Direct Preference Optimization: Your Language Model is Secretly a Reward Model](https://mlsanity.com/paper/2305.18290/) (2023)
- [LAION-5B: An open large-scale dataset for training next generation image-text models](https://mlsanity.com/paper/2210.08402/) (2022)
- [Deep reinforcement learning from human preferences](https://mlsanity.com/paper/1706.03741/) (2017)
- [Self-Instruct: Aligning Language Models with Self-Generated Instructions](https://mlsanity.com/paper/2212.10560/) (2022)
- [Don't Give Me the Details, Just the Summary! Topic-Aware Convolutional Neural Networks for Extreme Summarization](https://mlsanity.com/paper/1808.08745/) (2018)
- [Safe RLHF: Safe Reinforcement Learning from Human Feedback](https://mlsanity.com/paper/2310.12773/) (2023)
- [Train faster, generalize better: Stability of stochastic gradient descent](https://mlsanity.com/paper/1509.01240/) (2015)
- [Safety-Tuned LLaMAs: Lessons From Improving the Safety of Large Language Models that Follow Instructions](https://mlsanity.com/paper/2309.07875/) (2023)
- [Performative Prediction](https://mlsanity.com/paper/2002.06673/) (2020)
- [Influence Functions in Deep Learning Are Fragile](https://mlsanity.com/paper/2006.14651/) (2020)
- [Self-Consuming Generative Models Go MAD](https://mlsanity.com/paper/2307.01850/) (2023)
- [JourneyDB: A Benchmark for Generative Image Understanding](https://mlsanity.com/paper/2307.00716/) (2023)
- [The Lipschitz Constant of Self-Attention](https://mlsanity.com/paper/2006.04710/) (2020)
- [Is Model Collapse Inevitable? Breaking the Curse of Recursion by Accumulating Real and Synthetic Data](https://mlsanity.com/paper/2404.01413/) (2024)
- [On the Stability of Iterative Retraining of Generative Models on their own Data](https://mlsanity.com/paper/2310.00429/) (2023)
- [Performative Prediction in a Stateful World](https://mlsanity.com/paper/2011.03885/) (2020)
- [CapsFusion: Rethinking Image-Text Data at Scale](https://mlsanity.com/paper/2310.20550/) (2023)
- [How Bad is Training on Synthetic Data? A Statistical Analysis of Language Model Collapse](https://mlsanity.com/paper/2404.05090/) (2024)
- [CoEdIT: Text Editing by Task-Specific Instruction Tuning](https://mlsanity.com/paper/2305.09857/) (2023)
- [Strong Model Collapse](https://mlsanity.com/paper/2410.04840/) (2024)
- [A Dynamical System Perspective for Lipschitz Neural Networks](https://mlsanity.com/paper/2110.12690/) (2021)
- [Data Feedback Loops: Model-driven Amplification of Dataset Biases](https://mlsanity.com/paper/2209.03942/) (2022)
- [Fairness Feedback Loops: Training on Synthetic Data Amplifies Bias](https://mlsanity.com/paper/2403.07857/) (2024)
- [Distort-and-Recover: Color Enhancement using Deep Reinforcement Learning](https://mlsanity.com/paper/1804.04450/) (2018)
- [Beyond Model Collapse: Scaling Up with Synthesized Data Requires Verification](https://mlsanity.com/paper/2406.07515/) (2024)
- [Self-Consuming Generative Models with Curated Data Provably Optimize Human Preferences](https://mlsanity.com/paper/2407.09499/) (2024)

## Built on by (most cited first, 2 of 2)

- [Stability and Diversity of Networked Self-Consuming Generative Ecosystems](https://mlsanity.com/paper/2610.09409/) (2026)
- [Self-Consuming Generative Models with Co-Evolving Human Preferences](https://mlsanity.com/paper/2610.09415/) (2026)

## Abstract (arXiv)

Foundation models are increasingly trained on synthetic data generated by prior model iterations rather than exclusively on real data. This self-consuming training paradigm can lead to model collapse, divergence, or bias amplification. Recent work (Ferbach et al., 2024) shows that incorporating human curation into the loop can steer a self-consuming model toward human-aligned behavior, but these analyses focus on a single, isolated model that solely consumes its own outputs. In practice, however, models often interact and train on input-output pairs produced by other models. This paper studies self-consuming training in the multi-model regime. We first formalize a framework for interacting self-consuming models and characterize when the resulting dynamical system converges to a stable point. We then examine how human curation of one model affects its own alignment (self-influence) and how such effects propagate to other models (cross-influence). Unlike isolated settings where human curation always enhances model alignment, we show that cross-model interactions can dampen or even invert this effect, ultimately degrading long-term alignment.