mlsanityData to 7 Oct 2026

Efficient Diversity-Preserving Diffusion Alignment via Gradient-Informed GFlowNets

arXiv 2412.07775 ↗First posted 10 Dec 2024cs.LG cs.CVcited by 30

On the map: Diffusion models → Diffusion and flow policies for robotics

Abstract (arXiv)

While one commonly trains large diffusion models by collecting datasets on target downstream tasks, it is often desired to align and finetune pretrained diffusion models with some reward functions that are either designed by experts or learned from small-scale datasets. Existing post-training methods for reward finetuning of diffusion models typically suffer from lack of diversity in generated samples, lack of prior preservation, and/or slow convergence in finetuning. In response to this challenge, we take inspiration from recent successes in generative flow networks (GFlowNets) and propose a reinforcement learning method for diffusion model finetuning, dubbed Nabla-GFlowNet (abbreviated as nabla-GFlowNet), that leverages the rich signal in reward gradients for probabilistic diffusion finetuning. We show that our proposed method achieves fast yet diversity- and prior-preserving finetuning of Stable Diffusion, a large-scale text-conditioned image diffusion model, on different realistic reward functions.

Key facts. Cited by 30 papers in the map (data through 7 Oct 2026). Placed in Diffusion and flow policies for robotics, one of 14 directions in Diffusion models. Builds on 77 of its 91 references that are in the map. First posted on arXiv 10 Dec 2024.

Builds on

77 of 91 references.

Closest ancestors · 8

Same direction, newest first

  1. 2024Adjoint Matching: Fine-tuning Flow and Diffusion Generative Models with Memoryless Stochastic Optimal Control13 Sep
  2. 2024Improving GFlowNets for Text-to-Image Diffusion Alignment2 Jun
  3. 2024Fine-Tuning of Continuous-Time Diffusion Models as Entropy-Regularized Control23 Feb
  4. 2023Aligning Text-to-Image Diffusion Models with Reward Backpropagation5 Oct
  5. 2023Directly Fine-Tuning Diffusion Models on Differentiable Rewards29 Sep
  6. 2023DPOK: Reinforcement Learning for Fine-tuning Text-to-Image Diffusion Models25 May
  7. 2023Training Diffusion Models with Reinforcement Learning22 May
  8. 2023Human Preference Score: Better Aligning Text-to-Image Models with Human Preference25 Mar

Other references · 69

  1. 2020Denoising Diffusion Probabilistic Modelscited by 20,851
  2. 2015U-Net: Convolutional Networks for Biomedical Image Segmentationcited by 19,604
  3. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,694
  4. 2022Training language models to follow instructions with human feedbackcited by 15,712
  5. 2017Proximal Policy Optimization Algorithmscited by 14,441
  6. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,084
  7. 2020Denoising Diffusion Implicit Modelscited by 9,655
  8. 2020Score-Based Generative Modeling through Stochastic Differential Equationscited by 8,964
  9. 2021Diffusion Models Beat GANs on Image Synthesiscited by 8,185
  10. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 7,916
  11. 2022Photorealistic Text-to-Image Diffusion Models with Deep Language Understandingcited by 6,322
  12. 2022Flow Matching for Generative Modelingcited by 5,426
38 more
  1. 2018Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actorcited by 4,635
  2. 2017Deep reinforcement learning from human preferencescited by 3,846
  3. 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedbackcited by 3,634
  4. 2022DreamFusion: Text-to-3D using 2D Diffusioncited by 3,006
  5. 2020Learning to summarize from human feedbackcited by 2,562
  6. 2023Stable Video Diffusion: Scaling Latent Video Diffusion Models to Large Datasetscited by 2,526
  7. 2015Trust Region Policy Optimizationcited by 2,503
  8. 2022Video Diffusion Modelscited by 2,280
  9. 2019Fine-Tuning Language Models from Human Preferencescited by 2,124
  10. 2022Imagen Video: High Definition Video Generation with Diffusion Modelscited by 1,765
  11. 2023ImageReward: Learning and Evaluating Human Preferences for Text-to-Image Generationcited by 1,010
  12. 2022DPM-Solver++: Fast Solver for Guided Sampling of Diffusion Probabilistic Modelscited by 924
  13. 2023Human Preference Score v2: A Solid Benchmark for Evaluating Human Preferences of Text-to-Image Synthesiscited by 847
  14. 2023Discrete Diffusion Modeling by Estimating the Ratios of the Data Distributioncited by 712
  15. 2023RAFT: Reward rAnked FineTuning for Generative Foundation Model Alignmentcited by 672
  16. 2022Equivariant Diffusion for Molecule Generation in 3Dcited by 666
  17. 2017Reinforcement Learning with Deep Energy-Based Policiescited by 659
  18. 2022GeoDiff: a Geometric Diffusion Model for Molecular Conformation Generationcited by 511
  19. 2018Reinforcement Learning and Control as Probabilistic Inference: Tutorial and Reviewcited by 466
  20. 2024CLAY: A Controllable Large-scale Generative Model for Creating High-quality 3D Assetscited by 444
  21. 2023Aligning Text-to-Image Models using Human Feedbackcited by 418
  22. 2021Flow Network based Generative Models for Non-Iterative Diverse Candidate Generationcited by 384
  23. 2018Reward learning from human preferences and demonstrations in Ataricited by 314
  24. 2021GFlowNet Foundationscited by 286
  25. 2022Trajectory balance: Improved credit assignment in GFlowNetscited by 244
  26. 2023Practical and Asymptotically Exact Conditional Sampling in Diffusion Modelscited by 218
  27. 2023Controlling Text-to-Image Diffusion by Orthogonal Finetuningcited by 207
  28. 2023MeshDiffusion: Score-based Generative 3D Mesh Modelingcited by 198
  29. 2022Biological Sequence Design with GFlowNetscited by 172
  30. 2018Deep Reinforcement Learning and the Deadly Triadcited by 170
  31. 2022Bayesian Structure Learning with Generative Flow Networkscited by 153
  32. 2022Generative Flow Networks for Discrete Probabilistic Modelingcited by 113
  33. 2023Parameter-Efficient Orthogonal Finetuning via Butterfly Factorizationcited by 112
  34. 2022Latent Diffusion for Language Generationcited by 110
  35. 2022GFlowNets and variational inferencecited by 103
  36. 2023A theory of continuous generative flow networkscited by 98
  37. 2023GraphDreamer: Compositional 3D Scene Synthesis from Scene Graphscited by 96
  38. 2024Probabilistic Inference in Language Models via Twisted Sequential Monte Carlocited by 85

50 most cited of 69 All 69 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 1; 2025: 16; 2026: 13.

30 papers.

Sort:

By area:

  1. 2025FlowRL: Matching Reward Distributions for LLM Reasoningcited by 38
  2. 2025Trust Region Constrained Measure Transport in Path Space for Stochastic Optimal Control and Inferencecited by 22
  3. 2025Tilt Matching for Scalable Sampling and Fine-Tuningcited by 17
  4. 2026Reinforce Adjoint Matching: Scaling RL Post-Training of Diffusion and Flow-Matching Modelscited by 17
  5. 2025Learning to Sample Effective and Diverse Prompts for Text-to-Image Generationcited by 16
  6. 2025Value Gradient Guidance for Flow Matching Alignmentcited by 14
  7. 2025Inference-Time Scaling of Discrete Diffusion Models via Importance Weighting and Optimal Proposal Designcited by 12
  8. 2025Nabla-R2D3: Effective and Efficient 3D Diffusion Alignment with 2D Rewardscited by 11
  9. 2025CREPE: Controlling Diffusion with Replica Exchangecited by 9
  10. 2025EraseFlow: Learning Concept Erasure Policies via GFlowNet-Driven Alignmentcited by 8
  11. 2024Generating Attribution Reports for Manipulated Facial Images: A Dataset and Baselinecited by 6
  12. 2025Learning Flexible Forward Trajectories for Masked Molecular Diffusioncited by 6
18 more
  1. 2026Reward Score Matching: Unifying Reward-based Fine-tuning for Flow and Diffusion Modelscited by 5
  2. 2026Finite Difference Flow Optimization for RL Post-Training of Text-to-Image Modelscited by 4
  3. 2026TMPO: Trajectory Matching Policy Optimization for Diverse and Efficient Diffusion Alignmentcited by 4
  4. 2025VARD: Efficient and Dense Fine-Tuning for Diffusion Models with Value-based RLcited by 3
  5. 2025Concept-TRAK: Understanding how diffusion models learn concepts through concept-level attributioncited by 3
  6. 2025Mitigating Hallucinations in LM-Based TTS Models via Distribution Alignment Using GFlowNetscited by 2
  7. 2025Diffusion Alignment as Variational Expectation-Maximizationcited by 2
  8. 2025RMLer: Synthesizing Novel Objects across Diverse Categories via Reinforcement Mixing Learningcited by 1
  9. 2026Diffusion Alignment Beyond KL: Variance Minimisation as Effective Policy Optimisercited by 1
  10. 2026Rethinking Preference Alignment for Diffusion Models with Classifier-Free Guidancecited by 1
  11. 2026Gradient-Free Noise Optimization for Reward Alignment in Generative Modelscited by 1
  12. 2025Beyond the Proxy: Trajectory-Distilled Guidance for Offline GFlowNet Trainingcited by 0
  13. 2026Signal from Structure: Exploiting Submodular Upper Bounds in Generative Flow Networkscited by 0
  14. 2026Controlling Exploration-Exploitation in GFlowNets via Markov Chain Perspectivescited by 0
  15. 2026TILDE: TILt-based Distributional Erasure for Concept Unlearningcited by 0
  16. 2026Unified Trajectory Matching Policy Optimization: Diverse T2I Generation and VLA Generalizationcited by 0
  17. 2026ExploreNet: Learning Where to Explore in Diffusion GRPOcited by 0
  18. 2026Manifold-Constrained Initial Noise Optimization for Efficient Generative Model Alignmentcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025CoCA: Step-level Reward for Free in RL-based T2I Diffusion Model Fine-tuning
  2. 2025Diff2Flow: Training Flow Matching Models via Diffusion Model Alignment
  3. 2026Alignment of Diffusion Model and Flow Matching for Text-to-Image Generation
  4. 2025GARDO: Reinforcing Diffusion Models without Reward Hacking
  5. 2024Video Diffusion Alignment via Reward Gradients
  6. 2025Accelerating Diffusion Model Training under Minimal Budgets: A Condensation-Based Perspective
  7. 2025DiffusionNFT: Online Diffusion Reinforcement with Forward Process
  8. 2024Large-scale Reinforcement Learning for Diffusion Models