mlsanityData to 7 Oct 2026

Self-Play Fine-Tuning of Diffusion Models for Text-to-Image Generation

arXiv 2402.10210 ↗First posted 15 Feb 2024cs.LG cs.AI cs.CLcited by 85

On the map: Diffusion models → Diffusion and flow policies for robotics

also in 2 more
Abstract (arXiv)

Fine-tuning Diffusion Models remains an underexplored frontier in generative artificial intelligence (GenAI), especially when compared with the remarkable progress made in fine-tuning Large Language Models (LLMs). While cutting-edge diffusion models such as Stable Diffusion (SD) and SDXL rely on supervised fine-tuning, their performance inevitably plateaus after seeing a certain volume of data. Recently, reinforcement learning (RL) has been employed to fine-tune diffusion models with human preference data, but it requires at least two images ("winner" and "loser" images) for each text prompt. In this paper, we introduce an innovative technique called self-play fine-tuning for diffusion models (SPIN-Diffusion), where the diffusion model engages in competition with its earlier versions, facilitating an iterative self-improvement process. Our approach offers an alternative to conventional supervised fine-tuning and RL strategies, significantly improving both model performance and alignment. Our experiments on the Pick-a-Pic dataset reveal that SPIN-Diffusion outperforms the existing supervised fine-tuning method in aspects of human preference alignment and visual appeal right from its first iteration. By the second iteration, it exceeds the performance of RLHF-based methods across all metrics, achieving these results with less data.

Key facts. Cited by 85 papers in the map (data through 7 Oct 2026). Placed in Diffusion and flow policies for robotics, one of 14 directions in Diffusion models. Builds on 40 of its 49 references that are in the map. First posted on arXiv 15 Feb 2024.

Builds on

40 of 49 references.

Closest ancestors · 6

Same direction, newest first

  1. 2023Using Human Feedback to Fine-tune Diffusion Models without Any Reward Model22 Nov
  2. 2023Diffusion Model Alignment Using Direct Preference Optimization21 Nov
  3. 2023Aligning Text-to-Image Diffusion Models with Reward Backpropagation5 Oct
  4. 2023Directly Fine-Tuning Diffusion Models on Differentiable Rewards29 Sep
  5. 2023DPOK: Reinforcement Learning for Fine-tuning Text-to-Image Diffusion Models25 May
  6. 2023Training Diffusion Models with Reinforcement Learning22 May

Other references · 34

  1. 2020Denoising Diffusion Probabilistic Modelscited by 20,851
  2. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,694
  3. 2014Microsoft COCO: Common Objects in Contextcited by 17,498
  4. 2020Denoising Diffusion Implicit Modelscited by 9,655
  5. 2020Score-Based Generative Modeling through Stochastic Differential Equationscited by 8,964
  6. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 7,916
  7. 2015Deep Unsupervised Learning using Nonequilibrium Thermodynamicscited by 7,005
  8. 2022Hierarchical Text-Conditional Image Generation with CLIP Latentscited by 6,539
  9. 2022Photorealistic Text-to-Image Diffusion Models with Deep Language Understandingcited by 6,322
  10. 2022Scalable Diffusion Models with Transformerscited by 5,931
  11. 2023SDXL: Improving Latent Diffusion Models for High-Resolution Image Synthesiscited by 4,290
  12. 2022LAION-5B: An open large-scale dataset for training next generation image-text modelscited by 4,131
22 more
  1. 2019Generative Modeling by Estimating Gradients of the Data Distributioncited by 4,067
  2. 2021GLIDE: Towards Photorealistic Image Generation and Editing with Text-Guided Diffusion Modelscited by 3,595
  3. 2022DreamBooth: Fine Tuning Text-to-Image Diffusion Models for Subject-Driven Generationcited by 3,286
  4. 2022An Image is Worth One Word: Personalizing Text-to-Image Generation using Textual Inversioncited by 2,231
  5. 2021Structured Denoising Diffusion Models in Discrete State-Spacescited by 1,665
  6. 2021Image Super-Resolution via Iterative Refinementcited by 1,506
  7. 2022Scaling Autoregressive Models for Content-Rich Text-to-Image Generationcited by 1,287
  8. 2021Cascaded Diffusion Models for High Fidelity Image Generationcited by 1,155
  9. 2023ImageReward: Learning and Evaluating Human Preferences for Text-to-Image Generationcited by 1,010
  10. 2023Pick-a-Pic: An Open Dataset of User Preferences for Text-to-Image Generationcited by 950
  11. 2016COCO-Stuff: Thing and Stuff Classes in Contextcited by 858
  12. 2023Human Preference Score v2: A Solid Benchmark for Evaluating Human Preferences of Text-to-Image Synthesiscited by 847
  13. 2017Mastering Chess and Shogi by Self-Play with a General Reinforcement Learning Algorithmcited by 723
  14. 2017Generative Adversarial Networks: An Overviewcited by 588
  15. 2024Self-Play Fine-Tuning Converts Weak Language Models to Strong Language Modelscited by 552
  16. 2023Aligning Text-to-Image Models using Human Feedbackcited by 418
  17. 2022DiffDock: Diffusion Steps, Twists, and Turns for Molecular Dockingcited by 307
  18. 2023Emu: Enhancing Image Generation Models Using Photogenic Needles in a Haystackcited by 278
  19. 2023A Reparameterized Discrete Diffusion Model for Text Generationcited by 152
  20. 2021Learning to Efficiently Sample from Diffusion Probabilistic Modelscited by 129
  21. 2023A Picture is Worth a Thousand Words: Principled Recaptioning Improves Image Generationcited by 40
  22. 2024Parrot: Pareto-optimal Multi-Reward Reinforcement Learning Framework for Text-to-Image Generationcited by 35

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 14; 2025: 42; 2026: 29.

85 papers.

Sort:

By area:

  1. 2025Flow-GRPO: Training Flow Matching Models via Online RLcited by 605
  2. 2025Improving Video Generation with Human Feedbackcited by 267
  3. 2025DiffusionNFT: Online Diffusion Reinforcement with Forward Processcited by 204
  4. 2025Unified Reward Model for Multimodal Understanding and Generationcited by 194
  5. 2025MixGRPO: Unlocking Flow-based GRPO Efficiency with Mixed ODE-SDEcited by 168
  6. 2025X-Omni: Reinforcement Learning Makes Discrete Autoregressive Image Generative Models Great Againcited by 105
  7. 2025BranchGRPO: Stable and Efficient GRPO with Structured Branching in Diffusion Modelscited by 59
  8. 2024Inference-Time Alignment of Diffusion Models with Direct Noise Optimizationcited by 51
  9. 2025The Best Instruction-Tuning Data are Those That Fitcited by 48
  10. 2024SpeechAlign: Aligning Speech Generation to Human Preferencescited by 45
  11. 2024Margin-aware Preference Optimization for Aligning Diffusion Models without Referencecited by 45
  12. 2024Improving Dynamic Object Interactions in Text-to-Video Generation with AI Feedbackcited by 45
38 more
  1. 2025Inference-Time Text-to-Video Alignment with Diffusion Latent Beam Searchcited by 40
  2. 2025Calibrated Multi-Preference Optimization for Aligning Diffusion Modelscited by 40
  3. 2025Score as Action: Fine-Tuning Diffusion Generative Models by Continuous-time Reinforcement Learningcited by 33
  4. 2025NoHumansRequired: Autonomous High-Quality Image Editing Triplet Miningcited by 33
  5. 2025DenseDPO: Fine-Grained Temporal Preference Optimization for Video Diffusion Modelscited by 32
  6. 2025Fine-Grained Preference Optimization Improves Spatial Reasoning in VLMscited by 30
  7. 2024Tuning Timestep-Distilled Diffusion Model Using Pairwise Sample Optimizationcited by 25
  8. 2025Improving the Scaling Laws of Synthetic Data with Deliberate Practicecited by 17
  9. 2025GARDO: Reinforcing Diffusion Models without Reward Hackingcited by 17
  10. 2024Boost Your Human Image Generation Model via Direct Preference Optimizationcited by 16
  11. 2024SELMA: Learning and Merging Skill-Specific Text-to-Image Experts with Auto-Generated Datacited by 14
  12. 2025TR2-D2: Tree Search Guided Trajectory-Aware Fine-Tuning for Discrete Diffusioncited by 13
  13. 2024Aligning Few-Step Diffusion Models with Dense Reward Difference Learningcited by 11
  14. 2025The Image as Its Own Reward: Reinforcement Learning with Adversarial Reward for Image Generationcited by 11
  15. 2026UniGRPO: Unified Policy Optimization for Reasoning-Driven Visual Generationcited by 11
  16. 2024ByteEdit: Boost, Comply and Accelerate Generative Image Editingcited by 10
  17. 2024Text-to-Image Synthesis: A Decade Surveycited by 9
  18. 2025Identity-GRPO: Optimizing Multi-Human Identity-preserving Video Generation via Reinforcement Learningcited by 9
  19. 2026Triplets Better Than Pairs: Towards Stable and Effective Self-Play Fine-Tuning for LLMscited by 9
  20. 2024Information Theoretic Text-to-Image Alignmentcited by 8
  21. 2024Bridging SFT and DPO for Diffusion Model Alignment with Self-Sampling Preference Optimizationcited by 8
  22. 2025PiSA: A Self-Augmented Data Engine and Training Strategy for 3D Understanding with Large Modelscited by 8
  23. 2025Aligning Anime Video Generation with Human Feedbackcited by 8
  24. 2026TAGRPO: Boosting GRPO on Image-to-Video Generation with Direct Trajectory Alignmentcited by 8
  25. 2025MIRA: Towards Mitigating Reward Hacking in Inference-Time Alignment of T2I Diffusion Modelscited by 7
  26. 2025PaCo-RL: Advancing Reinforcement Learning for Consistent Image Generation with Pairwise Reward Modelingcited by 7
  27. 2025SPACE: Noise Contrastive Estimation Stabilizes Self-Play Fine-Tuning for Large Language Modelscited by 7
  28. 2024MoDiPO: text-to-motion alignment via AI-feedback-driven Direct Preference Optimizationcited by 6
  29. 2025RAPID^3: Tri-Level Reinforced Acceleration Policies for Diffusion Transformercited by 6
  30. 2026LeapAlign: Post-Training Flow Matching Models at Any Generation Step by Building Two-Step Trajectoriescited by 6
  31. 2024RFSR: Improving ISR Diffusion Models via Reward Feedback Learningcited by 5
  32. 2025Half-order Fine-Tuning for Diffusion Model: A Recursive Likelihood Ratio Optimizercited by 5
  33. 2025Neon: Negative Extrapolation From Self-Training Improves Image Generationcited by 5
  34. 2025IC-World: In-Context Generation for Shared World Modelingcited by 5
  35. 2025Anchoring Values in Temporal and Group Dimensions for Flow Matching Model Alignmentcited by 5
  36. 2025MultiCrafter: High-Fidelity Multi-Subject Generation via Disentangled Attention and Identity-Aware Preference Alignmentcited by 4
  37. 2025PCPO: Proportionate Credit Policy Optimization for Aligning Image Generation Modelscited by 4
  38. 2026Lens: Rethinking Training Efficiency for Foundational Text-to-Image Modelscited by 4

50 most cited of 85 All 85 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2024Large-scale Reinforcement Learning for Diffusion Models
  2. 2026CRAFT: Aligning Diffusion Models with Fine-Tuning Is Easier Than You Think
  3. 2025Alignment and Safety of Diffusion Models via Reinforcement Learning and Reward Modeling: A Survey
  4. 2025Diffusion Blend: Inference-Time Multi-Preference Alignment for Diffusion Models
  5. 2025A Simple and Effective Reinforcement Learning Method for Text-to-Image Diffusion Fine-tuning
  6. 2025Self-NPO: Data-Free Diffusion Model Enhancement via Truncated Diffusion Fine-Tuning
  7. 2024Understanding Reinforcement Learning-Based Fine-Tuning of Diffusion Models: A Tutorial and Review
  8. 2025Preference Alignment on Diffusion Model: A Comprehensive Survey for Image Generation and Editing