mlsanityData to 7 Oct 2026

Improving GFlowNets for Text-to-Image Diffusion Alignment

arXiv 2406.00633 ↗First posted 2 Jun 2024cs.LG cs.AI cs.CVcited by 27

On the map: Diffusion models → Diffusion and flow policies for robotics

Abstract (arXiv)

Diffusion models have become the de-facto approach for generating visual data, which are trained to match the distribution of the training dataset. In addition, we also want to control generation to fulfill desired properties such as alignment to a text description, which can be specified with a black-box reward function. Prior works fine-tune pretrained diffusion models to achieve this goal through reinforcement learning-based algorithms. Nonetheless, they suffer from issues including slow credit assignment as well as low quality in their generated samples. In this work, we explore techniques that do not directly maximize the reward but rather generate high-reward images with relatively high probability -- a natural scenario for the framework of generative flow networks (GFlowNets). To this end, we propose the Diffusion Alignment with GFlowNet (DAG) algorithm to post-train diffusion models with black-box property functions. Extensive experiments on Stable Diffusion and various reward specifications corroborate that our method could effectively align large-scale text-to-image diffusion models with given reward information.

Key facts. Cited by 27 papers in the map (data through 7 Oct 2026). Placed in Diffusion and flow policies for robotics, one of 14 directions in Diffusion models. Builds on 62 of its 76 references that are in the map. First posted on arXiv 2 Jun 2024.

Builds on

62 of 76 references.

Closest ancestors · 9

Same direction, newest first

  1. 2024Deep Reward Supervisions for Tuning Text-to-Image Diffusion Models1 May
  2. 2024Fine-Tuning of Continuous-Time Diffusion Models as Entropy-Regularized Control23 Feb
  3. 2023Enhancing Diffusion Models with Text-Encoder Reinforcement Learning27 Nov
  4. 2023Using Human Feedback to Fine-tune Diffusion Models without Any Reward Model22 Nov
  5. 2023Diffusion Model Alignment Using Direct Preference Optimization21 Nov
  6. 2023Aligning Text-to-Image Diffusion Models with Reward Backpropagation5 Oct
  7. 2023Directly Fine-Tuning Diffusion Models on Differentiable Rewards29 Sep
  8. 2023DPOK: Reinforcement Learning for Fine-tuning Text-to-Image Diffusion Models25 May
  9. 2023Training Diffusion Models with Reinforcement Learning22 May

Other references · 53

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  2. 2020Denoising Diffusion Probabilistic Modelscited by 20,851
  3. 2015U-Net: Convolutional Networks for Biomedical Image Segmentationcited by 19,604
  4. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,694
  5. 2017Proximal Policy Optimization Algorithmscited by 14,441
  6. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,084
  7. 2020Score-Based Generative Modeling through Stochastic Differential Equationscited by 8,964
  8. 2021Diffusion Models Beat GANs on Image Synthesiscited by 8,185
  9. 2015Deep Unsupervised Learning using Nonequilibrium Thermodynamicscited by 7,005
  10. 2022Photorealistic Text-to-Image Diffusion Models with Deep Language Understandingcited by 6,322
  11. 2018Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actorcited by 4,635
  12. 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedbackcited by 3,634
38 more
  1. 2022DreamFusion: Text-to-3D using 2D Diffusioncited by 3,006
  2. 2022Planning with Diffusion for Flexible Behavior Synthesiscited by 1,023
  3. 2023ImageReward: Learning and Evaluating Human Preferences for Text-to-Image Generationcited by 1,010
  4. 2023Human Preference Score v2: A Solid Benchmark for Evaluating Human Preferences of Text-to-Image Synthesiscited by 847
  5. 2017Reinforcement Learning with Deep Energy-Based Policiescited by 659
  6. 2023Universal Guidance for Diffusion Modelscited by 426
  7. 2023Aligning Text-to-Image Models using Human Feedbackcited by 418
  8. 2021Flow Network based Generative Models for Non-Iterative Diverse Candidate Generationcited by 384
  9. 2018Reward learning from human preferences and demonstrations in Ataricited by 314
  10. 2021GFlowNet Foundationscited by 286
  11. 2022Trajectory balance: Improved credit assignment in GFlowNetscited by 244
  12. 2022Biological Sequence Design with GFlowNetscited by 172
  13. 2022Bayesian Structure Learning with Generative Flow Networkscited by 153
  14. 2022Learning GFlowNets from partial episodes for improved convergence and stabilitycited by 129
  15. 2015Taming the Noise in Reinforcement Learning via Soft Updatescited by 125
  16. 2022Generative Flow Networks for Discrete Probabilistic Modelingcited by 113
  17. 2023End-to-End Diffusion Latent Optimization Improves Classifier Guidancecited by 107
  18. 2022GFlowNets and variational inferencecited by 103
  19. 2022Multi-Objective GFlowNetscited by 100
  20. 2023A theory of continuous generative flow networkscited by 98
  21. 2023Better Training of GFlowNets with Local Credit and Incomplete Trajectoriescited by 83
  22. 2023Towards Understanding and Improving GFlowNet Trainingcited by 74
  23. 2024Amortizing intractable inference in diffusion models for vision, language, and controlcited by 70
  24. 2023Robust Scheduling with GFlowNetscited by 67
  25. 2023GFlowNets for AI-Driven Scientific Discoverycited by 63
  26. 2023Generative Flow Networks as Entropy-Regularized RLcited by 60
  27. 2023Diffusion Generative Flow Samplers: Improving learning signals through partial trajectory optimizationcited by 57
  28. 2023Scalable Diffusion for Materials Generationcited by 52
  29. 2022Generative Augmented Flow Networkscited by 48
  30. 2023Local Search GFlowNetscited by 45
  31. 2023Let the Flows Tell: Solving Graph Combinatorial Optimization Problems with GFlowNetscited by 40
  32. 2024Discrete Probabilistic Inference as Control in Multi-path Environmentscited by 40
  33. 2022A Variational Perspective on Generative Flow Networkscited by 39
  34. 2024Diffusion Models as Constrained Samplers for Optimization with Unknown Constraintscited by 33
  35. 2023Stochastic Generative Flow Networkscited by 32
  36. 2023Maximum entropy GFlowNets with soft Q-learningcited by 30
  37. 2023Distributional GFlowNets with Quantile Flowscited by 28
  38. 2023Learning to Scale Logits for Temperature-Conditional GFlowNetscited by 26

50 most cited of 53 All 53 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 5; 2025: 13; 2026: 9.

27 papers.

Sort:

By area:

  1. 2024Adjoint Matching: Fine-tuning Flow and Diffusion Generative Models with Memoryless Stochastic Optimal Controlcited by 222
  2. 2025FlowRL: Matching Reward Distributions for LLM Reasoningcited by 38
  3. 2024Efficient Diversity-Preserving Diffusion Alignment via Gradient-Informed GFlowNetscited by 30
  4. 2025Trust Region Constrained Measure Transport in Path Space for Stochastic Optimal Control and Inferencecited by 22
  5. 2024Reward-Directed Score-Based Diffusion Models via q-Learningcited by 18
  6. 2025Tilt Matching for Scalable Sampling and Fine-Tuningcited by 17
  7. 2025Learning to Sample Effective and Diverse Prompts for Text-to-Image Generationcited by 16
  8. 2025Generative Models in Decision Making: A Surveycited by 14
  9. 2025Value Gradient Guidance for Flow Matching Alignmentcited by 14
  10. 2024Optimizing Backward Policies in GFlowNets via Trajectory Likelihood Maximizationcited by 12
  11. 2025Nabla-R2D3: Effective and Efficient 3D Diffusion Alignment with 2D Rewardscited by 11
  12. 2025CREPE: Controlling Diffusion with Replica Exchangecited by 9
15 more
  1. 2025Revisiting Non-Acyclic GFlowNets in Discrete Environmentscited by 8
  2. 2025EraseFlow: Learning Concept Erasure Policies via GFlowNet-Driven Alignmentcited by 8
  3. 2024Baking Symmetry into GFlowNetscited by 7
  4. 2026Reward Score Matching: Unifying Reward-based Fine-tuning for Flow and Diffusion Modelscited by 5
  5. 2025AbFlowNet: Optimizing Antibody-Antigen Binding Energy via Diffusion-GFlowNet Fusioncited by 4
  6. 2025Loss-Guided Auxiliary Agents for Overcoming Mode Collapse in GFlowNetscited by 4
  7. 2026Finite Difference Flow Optimization for RL Post-Training of Text-to-Image Modelscited by 4
  8. 2026TMPO: Trajectory Matching Policy Optimization for Diverse and Efficient Diffusion Alignmentcited by 4
  9. 2026Improved techniques for fine-tuning flow models via adjoint matching: a deterministic control pipelinecited by 3
  10. 2025Mitigating Hallucinations in LM-Based TTS Models via Distribution Alignment Using GFlowNetscited by 2
  11. 2026Critic-Guided Reinforcement Unlearning in Text-to-Image Diffusioncited by 1
  12. 2026Diffusion Alignment Beyond KL: Variance Minimisation as Effective Policy Optimisercited by 1
  13. 2026Your GFlowNet Secretly Learns an Optimal Transport Plancited by 0
  14. 2026Unified Trajectory Matching Policy Optimization: Diverse T2I Generation and VLA Generalizationcited by 0
  15. 2026Manifold-Constrained Initial Noise Optimization for Efficient Generative Model Alignmentcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026Alignment of Diffusion Model and Flow Matching for Text-to-Image Generation
  2. 2024Large-scale Reinforcement Learning for Diffusion Models
  3. 2025Alignment and Safety of Diffusion Models via Reinforcement Learning and Reward Modeling: A Survey
  4. 2025Towards Better Alignment: Training Diffusion Models with Reinforcement Learning Against Sparse Rewards
  5. 2026Alignment-Guided Score Matching for Text-to-Image Alignment in Diffusion Models
  6. 2025GARDO: Reinforcing Diffusion Models without Reward Hacking
  7. 2025Fine-tuning Flow Matching Generative Models with Intermediate Feedback
  8. 2024Video Diffusion Alignment via Reward Gradients