mlsanityData to 7 Oct 2026

Feedback Efficient Online Fine-Tuning of Diffusion Models

arXiv 2402.16359 ↗First posted 26 Feb 2024cs.LG cs.AI q-bio.QMcited by 52

On the map: Diffusion models → Diffusion and flow policies for robotics

Abstract (arXiv)

Diffusion models excel at modeling complex data distributions, including those of images, proteins, and small molecules. However, in many cases, our goal is to model parts of the distribution that maximize certain properties: for example, we may want to generate images with high aesthetic quality, or molecules with high bioactivity. It is natural to frame this as a reinforcement learning (RL) problem, in which the objective is to fine-tune a diffusion model to maximize a reward function that corresponds to some property. Even with access to online queries of the ground-truth reward function, efficiently discovering high-reward samples can be challenging: they might have a low probability in the initial distribution, and there might be many infeasible samples that do not even have a well-defined reward (e.g., unnatural images or physically impossible molecules). In this work, we propose a novel reinforcement learning procedure that efficiently explores on the manifold of feasible samples. We present a theoretical analysis providing a regret guarantee, as well as empirical validation across three domains: images, biological sequences, and molecules.

Key facts. Cited by 52 papers in the map (data through 7 Oct 2026). Placed in Diffusion and flow policies for robotics, one of 14 directions in Diffusion models. Builds on 59 of its 78 references that are in the map. First posted on arXiv 26 Feb 2024.

Builds on

59 of 78 references.

Closest ancestors · 9

Same direction, newest first

  1. 2024Fine-Tuning of Continuous-Time Diffusion Models as Entropy-Regularized Control23 Feb
  2. 2023Using Human Feedback to Fine-tune Diffusion Models without Any Reward Model22 Nov
  3. 2023Diffusion Model Alignment Using Direct Preference Optimization21 Nov
  4. 2023Aligning Text-to-Image Diffusion Models with Reward Backpropagation5 Oct
  5. 2023Directly Fine-Tuning Diffusion Models on Differentiable Rewards29 Sep
  6. 2023Reward-Directed Conditional Diffusion: Provable Distribution Estimation and Reward Improvement13 Jul
  7. 2023DPOK: Reinforcement Learning for Fine-tuning Text-to-Image Diffusion Models25 May
  8. 2023Training Diffusion Models with Reinforcement Learning22 May
  9. 2022Diffusion Policies as an Expressive Policy Class for Offline Reinforcement Learning12 Aug

Other references · 50

  1. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  2. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  3. 2020Denoising Diffusion Probabilistic Modelscited by 20,851
  4. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,694
  5. 2022Training language models to follow instructions with human feedbackcited by 15,712
  6. 2017Proximal Policy Optimization Algorithmscited by 14,441
  7. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,084
  8. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  9. 2020Denoising Diffusion Implicit Modelscited by 9,655
  10. 2021Diffusion Models Beat GANs on Image Synthesiscited by 8,185
  11. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 7,916
  12. 2022Flow Matching for Generative Modelingcited by 5,426
38 more
  1. 2013Auto-Encoding Variational Bayescited by 3,980
  2. 2023ImageReward: Learning and Evaluating Human Preferences for Text-to-Image Generationcited by 1,010
  3. 2023Stochastic Interpolants: A Unifying Framework for Flows and Diffusionscited by 897
  4. 2023Improving and generalizing flow-based generative models with minibatch optimal transportcited by 835
  5. 2016Training Deep Nets with Sublinear Memory Costcited by 825
  6. 2016Automatic chemical design using a data-driven continuous representation of moleculescited by 762
  7. 2018Deep Reinforcement Learning in a Handful of Trials using Probabilistic Dynamics Modelscited by 748
  8. 2018A Tutorial on Bayesian Optimizationcited by 734
  9. 2023RAFT: Reward rAnked FineTuning for Generative Foundation Model Alignmentcited by 672
  10. 2016Deep Exploration via Bootstrapped DQNcited by 518
  11. 2022DiGress: Discrete Denoising diffusion for graph generationcited by 503
  12. 2023Universal Guidance for Diffusion Modelscited by 426
  13. 2023Aligning Text-to-Image Models using Human Feedbackcited by 418
  14. 2018Neural Processescited by 307
  15. 2019A Theory of Regularized Markov Decision Processescited by 279
  16. 2022Score-based Generative Modeling of Graphs via the System of Stochastic Differential Equationscited by 273
  17. 2022Torsional Diffusion for Molecular Conformer Generationcited by 262
  18. 2022Diffusion models as plug-and-play priorscited by 230
  19. 2013Finite-Time Analysis of Kernelised Contextual Banditscited by 177
  20. 2017A unified view of entropy-regularized Markov decision processescited by 174
  21. 2023Protein Design with Guided Discrete Diffusioncited by 154
  22. 2022Equivariant 3D-Conditional Diffusion Models for Molecular Linker Designcited by 121
  23. 2022Let us Build Bridges: Understanding and Extending Diffusion Generative Modelscited by 102
  24. 2019Batched Multi-armed Bandits Problemcited by 97
  25. 2022Design-Bench: Benchmarks for Data-Driven Offline Model-Based Optimizationcited by 96
  26. 2016Memory-Efficient Backpropagation Through Timecited by 78
  27. 2023Diffusion Models for Black-Box Optimizationcited by 75
  28. 2023Dirichlet Diffusion Score Model for Biological Sequence Generationcited by 73
  29. 2021Improving black-box optimization in VAE latent space using decoder uncertaintycited by 51
  30. 2022Making Linear MDPs Practical via Contrastive Representation Learningcited by 49
  31. 2019Contextual Bandits with Continuous Actions: Smoothing, Zooming, and Adaptingcited by 42
  32. 2022A Survey of Active Learning for Natural Language Processingcited by 40
  33. 2018Garbage In, Reward Out: Bootstrapping Exploration in Multi-Armed Banditscited by 33
  34. 2021Mitigating Covariate Shift in Imitation Learning via Offline Data Without Great Coveragecited by 26
  35. 2022Contrastive UCB: Provably Efficient Contrastive Self-Supervised Learning in Online Reinforcement Learningcited by 26
  36. 2023Pre-Training Protein Encoder via Siamese Sequence-Structure Diffusion Trajectory Predictioncited by 13
  37. 2023Latent Diffusion Model for DNA Sequence Generationcited by 8
  38. 2023GraphGUIDE: interpretable and controllable conditional graph generation with discrete Bernoulli diffusioncited by 6

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 14; 2025: 23; 2026: 15.

52 papers.

Sort:

By area:

  1. 2025DexVLA: Vision-Language Model with Plug-In Diffusion Expert for General Robot Controlcited by 191
  2. 2025ChatVLA: Unified Multimodal Understanding and Robot Control with Vision-Language-Action Modelcited by 112
  3. 2024Understanding Reinforcement Learning-Based Fine-Tuning of Diffusion Models: A Tutorial and Reviewcited by 92
  4. 2025Test-time Alignment of Diffusion Models without Reward Over-optimizationcited by 89
  5. 2024An Overview of Diffusion Models: Applications, Guided Generation, Statistical Rates and Optimizationcited by 72
  6. 2024Diffusion-VLA: Generalizable and Interpretable Robot Foundation Model via Self-Generated Reasoningcited by 57
  7. 2024Inference-Time Alignment of Diffusion Models with Direct Noise Optimizationcited by 51
  8. 2025ChatVLA-2: Vision-Language-Action Model with Open-World Embodied Reasoning from Pretrained Knowledgecited by 49
  9. 2024Alignment of Diffusion Models: Fundamentals, Challenges, and Futurecited by 34
  10. 2024Aligning Target-Aware Molecule Diffusion Models with Exact Energy Optimizationcited by 25
  11. 2025Reward-Guided Iterative Refinement in Diffusion Models at Test-Time with Applications to Protein and DNA Designcited by 21
  12. 2024Stochastic Control for Fine-tuning Diffusion Models: Optimality, Regularity, and Convergencecited by 18
38 more
  1. 2024Adding Conditional Control to Diffusion Models with Reinforcement Learningcited by 16
  2. 2025Flow Density Control: Generative Optimization Beyond Entropy-Regularized Fine-Tuningcited by 15
  3. 2025Inference-Time Scaling of Discrete Diffusion Models via Importance Weighting and Optimal Proposal Designcited by 12
  4. 2025Provable Maximum Entropy Manifold Exploration via Diffusion Modelscited by 12
  5. 2024Aligning Few-Step Diffusion Models with Dense Reward Difference Learningcited by 11
  6. 2025Composition and Alignment of Diffusion Models using Constrained Learningcited by 8
  7. 2025Diffusion-Sharpening: Fine-tuning Diffusion Models with Denoising Trajectory Sharpeningcited by 6
  8. 2025Inversion-DPO: Precise and Efficient Post-Training for Diffusion Modelscited by 6
  9. 2026Verifier-Constrained Flow Expansion for Discovery Beyond the Datacited by 6
  10. 2026Efficient Tail-Aware Generative Optimization via Flow Model Fine-Tuningcited by 6
  11. 2025Diffusion Fine-Tuning via Reparameterized Policy Gradient of the Soft Q-Functioncited by 5
  12. 2024Maximum Entropy Inverse Reinforcement Learning of Diffusion Models with Energy-Based Modelscited by 4
  13. 2024Cell Morphology-Guided Small Molecule Generation with GFlowNetscited by 4
  14. 2024HERO: Human-Feedback Efficient Reinforcement Learning for Online Diffusion Model Finetuningcited by 4
  15. 2024Enhancing Exploration with Diffusion Policies in Hybrid Off-Policy RL: Application to Non-Prehensile Manipulationcited by 3
  16. 2025VARD: Efficient and Dense Fine-Tuning for Diffusion Models with Value-based RLcited by 3
  17. 2025Alignment and Safety of Diffusion Models via Reinforcement Learning and Reward Modeling: A Surveycited by 3
  18. 2025Is RL fine-tuning harder than regression? A PDE learning approach for diffusion modelscited by 3
  19. 2025Direct Preference Optimization-Enhanced Multi-Guided Diffusion Model for Traffic Scenario Generationcited by 2
  20. 2025DiffExp: Efficient Exploration in Reward Fine-tuning for Text-to-Image Diffusion Modelscited by 2
  21. 2025Sample and Computationally Efficient Continuous-Time Reinforcement Learning with General Function Approximationcited by 2
  22. 2025Efficient Controllable Diffusion via Optimal Classifier Guidancecited by 2
  23. 2026Personalized Image Generation via Human-in-the-loop Bayesian Optimizationcited by 2
  24. 2026A Unified Density Operator View of Flow Control and Mergingcited by 2
  25. 2026Constrained Flow Optimization via Sequential Fine Tuning for Molecular Designcited by 2
  26. 2024Pixel-wise RL on Diffusion Models: Reinforcement Learning from Rich Feedbackcited by 1
  27. 2025PIRF: Physics-Informed Reward Fine-Tuning for Diffusion Modelscited by 1
  28. 2025Embodiment Transfer Learning for Vision-Language-Action Modelscited by 1
  29. 2026Beyond Binary Preference: Aligning Diffusion Models to Fine-grained Criteria by Decoupling Attributescited by 1
  30. 2026DAM-VLA: A Dynamic Action Model-Based Vision-Language-Action Framework for Robot Manipulationcited by 1
  31. 2026Support-Proximity Augmented Diffusion Estimation for Offline Black-Box Optimizationcited by 1
  32. 2026BFMT: Enhancing Search Capabilities of Tree Sampler via Bootstrap Flow-Map Treecited by 1
  33. 2025Robot Metacognition: Decision Making with Confidence for Tool Inventioncited by 0
  34. 2025Unified Control for Inference-Time Guidance of Denoising Diffusion Modelscited by 0
  35. 2026Adaptive Linear Path Model-Based Diffusioncited by 0
  36. 2026PAPA: Online Personalized Active Preference Alignmentcited by 0
  37. 2026Sequentially-Controlled Interactive Multi-Particle Flow-Maps for Online Feedback-Driven Searchcited by 0
  38. 2026On the Design Space of Discrete Diffusion Online Adaptation for Molecular Optimizationcited by 0

50 most cited of 52 All 52 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2025Iterative Distillation for Reward-Guided Fine-Tuning of Diffusion Models in Biomolecular Design
  2. 2023Towards Controllable Diffusion Models via Reward-Guided Exploration
  3. 2024Derivative-Free Guidance in Continuous and Discrete Diffusion Models with Soft Value-Based Decoding
  4. 2024Bridging Model-Based Optimization and Generative Modeling via Conservative Fine-Tuning of Diffusion Models
  5. 2024Large-scale Reinforcement Learning for Diffusion Models
  6. 2025CoCA: Step-level Reward for Free in RL-based T2I Diffusion Model Fine-tuning
  7. 2025ReDiF: Reinforced Distillation for Few Step Diffusion
  8. 2025Fine-Tuning Diffusion Models via Intermediate Distribution Shaping