mlsanityData to 14 Oct 2026

Fine-Tuning of Continuous-Time Diffusion Models as Entropy-Regularized Control

arXiv 2402.15194 ↗First posted 23 Feb 2024cs.LG cs.AI stat.MLcited by 143

On the map: Diffusion models → Diffusion and flow policies for robotics

Abstract (arXiv)

Diffusion models excel at capturing complex data distributions, such as those of natural images and proteins. While diffusion models are trained to represent the distribution in the training dataset, we often are more concerned with other properties, such as the aesthetic quality of the generated images or the functional properties of generated proteins. Diffusion models can be finetuned in a goal-directed way by maximizing the value of some reward function (e.g., the aesthetic quality of an image). However, these approaches may lead to reduced sample diversity, significant deviations from the training data distribution, and even poor sample quality due to the exploitation of an imperfect reward function. The last issue often occurs when the reward function is a learned model meant to approximate a ground-truth "genuine" reward, as is the case in many practical applications. These challenges, collectively termed "reward collapse," pose a substantial obstacle. To address this reward collapse, we frame the finetuning problem as entropy-regularized control against the pretrained diffusion model, i.e., directly optimizing entropy-enhanced rewards with neural SDEs. We present theoretical and empirical evidence that demonstrates our framework is capable of efficiently generating diverse samples with high genuine rewards, mitigating the overoptimization of imperfect reward models.

Key facts. Cited by 143 papers in the map (data through 14 Oct 2026). Placed in Diffusion and flow policies for robotics, one of 14 directions in Diffusion models. Builds on 53 of its 75 references that are in the map. First posted on arXiv 23 Feb 2024.

Builds on

53 of 75 references.

Closest ancestors · 5

Same direction, newest first

  1. 2023Aligning Text-to-Image Diffusion Models with Reward Backpropagation5 Oct
  2. 2023Directly Fine-Tuning Diffusion Models on Differentiable Rewards29 Sep
  3. 2023Reward-Directed Conditional Diffusion: Provable Distribution Estimation and Reward Improvement13 Jul
  4. 2023DPOK: Reinforcement Learning for Fine-tuning Text-to-Image Diffusion Models25 May
  5. 2023Training Diffusion Models with Reinforcement Learning22 May

Other references · 48

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  2. 2020Denoising Diffusion Probabilistic Modelscited by 21,076
  3. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,863
  4. 2022Training language models to follow instructions with human feedbackcited by 15,971
  5. 2017Proximal Policy Optimization Algorithmscited by 15,215
  6. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,380
  7. 2020Denoising Diffusion Implicit Modelscited by 9,781
  8. 2020Score-Based Generative Modeling through Stochastic Differential Equationscited by 9,122
  9. 2021Diffusion Models Beat GANs on Image Synthesiscited by 8,255
  10. 2023Visual Instruction Tuningcited by 7,869
  11. 2015Deep Unsupervised Learning using Nonequilibrium Thermodynamicscited by 7,058
  12. 2022Flow Matching for Generative Modelingcited by 5,702
36 more
  1. 2018Neural Ordinary Differential Equationscited by 3,779
  2. 2022Flow Straight and Fast: Learning to Generate and Transfer Data with Rectified Flowcited by 3,700
  3. 2022Constitutional AI: Harmlessness from AI Feedbackcited by 3,050
  4. 2020Learning to summarize from human feedbackcited by 2,588
  5. 2013Sinkhorn Distances: Lightspeed Computation of Optimal Transportation Distancescited by 2,316
  6. 2019Normalizing Flows for Probabilistic Modeling and Inferencecited by 1,323
  7. 2022Scaling Laws for Reward Model Overoptimizationcited by 1,095
  8. 2023ImageReward: Learning and Evaluating Human Preferences for Text-to-Image Generationcited by 1,045
  9. 2016Training Deep Nets with Sublinear Memory Costcited by 848
  10. 2023Open Problems and Fundamental Limitations of Reinforcement Learning from Human Feedbackcited by 759
  11. 2021Diffusion Schr"odinger Bridge with Applications to Score-Based Generative Modelingcited by 602
  12. 2018Reinforcement Learning and Control as Probabilistic Inference: Tutorial and Reviewcited by 482
  13. 2023Universal Guidance for Diffusion Modelscited by 437
  14. 2023Aligning Text-to-Image Models using Human Feedbackcited by 422
  15. 2021GFlowNet Foundationscited by 290
  16. 2020Scalable Gradients for Stochastic Differential Equationscited by 271
  17. 2022Torsional Diffusion for Molecular Conformer Generationcited by 263
  18. 2022Diffusion models as plug-and-play priorscited by 231
  19. 2023I2SB: Image-to-Image Schr"odinger Bridgecited by 218
  20. 2022Learning Fast Samplers for Diffusion Models by Differentiating Through Sample Qualitycited by 182
  21. 2021Path Integral Sampler: a stochastic control approach for samplingcited by 181
  22. 2023Inversion by Direct Iteration: An Alternative to Denoising Diffusion for Image Restorationcited by 165
  23. 2023Protein Design with Guided Discrete Diffusioncited by 157
  24. 2022An optimal control perspective on diffusion-based generative modelingcited by 147
  25. 2022Diffusion-based Molecule Generation with Informative Prior Bridgescited by 106
  26. 2022Let us Build Bridges: Understanding and Extending Diffusion Generative Modelscited by 103
  27. 2023A theory of continuous generative flow networkscited by 101
  28. 2019Theoretical guarantees for sampling and inference in generative models with latent diffusionscited by 97
  29. 2022Design-Bench: Benchmarks for Data-Driven Offline Model-Based Optimizationcited by 97
  30. 2016Memory-Efficient Backpropagation Through Timecited by 78
  31. 2023Dirichlet Diffusion Score Model for Biological Sequence Generationcited by 75
  32. 2021Efficient and Accurate Gradients for Neural SDEscited by 71
  33. 2018Sampling Can Be Faster Than Optimizationcited by 68
  34. 2023Diffusion Generative Flow Samplers: Improving learning signals through partial trajectory optimizationcited by 57
  35. 2024Implicit Diffusion: Efficient Optimization through Stochastic Samplingcited by 24
  36. 2019Schr"odinger Bridge Samplerscited by 14

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 29; 2025: 53; 2026: 61.

143 papers.

Sort:

By area:

  1. 2024Adjoint Matching: Fine-tuning Flow and Diffusion Generative Models with Memoryless Stochastic Optimal Controlcited by 233
  2. 2025DexVLA: Vision-Language Model with Plug-In Diffusion Expert for General Robot Controlcited by 195
  3. 2024Derivative-Free Guidance in Continuous and Discrete Diffusion Models with Soft Value-Based Decodingcited by 136
  4. 2025ChatVLA: Unified Multimodal Understanding and Robot Control with Vision-Language-Action Modelcited by 113
  5. 2024Understanding Reinforcement Learning-Based Fine-Tuning of Diffusion Models: A Tutorial and Reviewcited by 98
  6. 2025Test-time Alignment of Diffusion Models without Reward Over-optimizationcited by 93
  7. 2025Inference-Time Alignment in Diffusion Models with Reward-Guided Generation: Tutorial and Reviewcited by 88
  8. 2024Fine-Tuning Discrete Diffusion Models via Reward Optimization with Applications to DNA and Protein Designcited by 79
  9. 2024An Overview of Diffusion Models: Applications, Guided Generation, Statistical Rates and Optimizationcited by 73
  10. 2024Amortizing intractable inference in diffusion models for vision, language, and controlcited by 72
  11. 2024Gradient Guidance for Diffusion Models: An Optimization Perspectivecited by 61
  12. 2024Diffusion-VLA: Generalizable and Interpretable Robot Foundation Model via Self-Generated Reasoningcited by 57
38 more
  1. 2024Feedback Efficient Online Fine-Tuning of Diffusion Modelscited by 53
  2. 2025ChatVLA-2: Vision-Language-Action Model with Open-World Embodied Reasoning from Pretrained Knowledgecited by 49
  3. 2024Improving Dynamic Object Interactions in Text-to-Video Generation with AI Feedbackcited by 47
  4. 2024DEFT: Efficient Fine-Tuning of Diffusion Models by Learning the Generalised h-transformcited by 46
  5. 2024Fine-tuning of diffusion models via stochastic control: entropy regularization and beyondcited by 42
  6. 2025Inference-Time Text-to-Video Alignment with Diffusion Latent Beam Searchcited by 41
  7. 2025Scaling Image and Video Generation via Test-Time Evolutionary Searchcited by 40
  8. 2025Inference-Time Scaling for Flow Models via Stochastic Generation and Rollover Budget Forcingcited by 38
  9. 2024Alignment of Diffusion Models: Fundamentals, Challenges, and Futurecited by 35
  10. 2025Noise Hypernetworks: Amortizing Test-Time Compute in Diffusion Modelscited by 35
  11. 2025Diffusion Tree Sampling: Scalable inference-time alignment of diffusion modelscited by 34
  12. 2025Score as Action: Fine-Tuning Diffusion Generative Models by Continuous-time Reinforcement Learningcited by 33
  13. 2024Steering Masked Discrete Diffusion Models via Discrete Denoising Posterior Predictioncited by 32
  14. 2024Efficient Diversity-Preserving Diffusion Alignment via Gradient-Informed GFlowNetscited by 32
  15. 2026Diamond Maps: Efficient Reward Alignment via Stochastic Flow Mapscited by 29
  16. 2024Improving GFlowNets for Text-to-Image Diffusion Alignmentcited by 27
  17. 2024Bridging Model-Based Optimization and Generative Modeling via Conservative Fine-Tuning of Diffusion Modelscited by 25
  18. 2024Aligning Target-Aware Molecule Diffusion Models with Exact Energy Optimizationcited by 25
  19. 2025Reward-Guided Iterative Refinement in Diffusion Models at Test-Time with Applications to Protein and DNA Designcited by 25
  20. 2025Training-Free Guidance Beyond Differentiability: Scalable Path Steering with Tree Search in Diffusion and Flow Modelscited by 24
  21. 2024Preference Tuning with Human Feedback on Language, Speech, and Vision Tasks: A Surveycited by 23
  22. 2025Trust Region Constrained Measure Transport in Path Space for Stochastic Optimal Control and Inferencecited by 23
  23. 2025DriftLite: Lightweight Drift Control for Inference-Time Scaling of Diffusion Modelscited by 23
  24. 2024Constrained Diffusion Models via Dual Trainingcited by 20
  25. 2024Reward-Directed Score-Based Diffusion Models via q-Learningcited by 19
  26. 2024Scores as Actions: a framework of fine-tuning diffusion models by continuous-time reinforcement learningcited by 19
  27. 2026Reinforce Adjoint Matching: Scaling RL Post-Training of Diffusion and Flow-Matching Modelscited by 19
  28. 2025Outsourced diffusion sampling: Efficient posterior inference in latent spaces of generative modelscited by 17
  29. 2025Flow Density Control: Generative Optimization Beyond Entropy-Regularized Fine-Tuningcited by 17
  30. 2025GARDO: Reinforcing Diffusion Models without Reward Hackingcited by 17
  31. 2024Adding Conditional Control to Diffusion Models with Reinforcement Learningcited by 16
  32. 2025Psi-Sampler: Initial Particle Sampling for SMC-Based Inference-Time Reward Alignment in Score Modelscited by 15
  33. 2025Provable Maximum Entropy Manifold Exploration via Diffusion Modelscited by 14
  34. 2025TR2-D2: Tree Search Guided Trajectory-Aware Fine-Tuning for Discrete Diffusioncited by 14
  35. 2025Value Gradient Guidance for Flow Matching Alignmentcited by 14
  36. 2024A Taxonomy of Loss Functions for Stochastic Optimal Controlcited by 13
  37. 2025Posterior Inference with Diffusion Models for High-dimensional Black-box Optimizationcited by 13
  38. 2025Source-Guided Flow Matchingcited by 13

50 most cited of 143 All 143 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2025CoCA: Step-level Reward for Free in RL-based T2I Diffusion Model Fine-tuning
  2. 2023Towards Controllable Diffusion Models via Reward-Guided Exploration
  3. 2025Iterative Importance Fine-tuning of Diffusion Models
  4. 2025PIRF: Physics-Informed Reward Fine-Tuning for Diffusion Models
  5. 2026IDRF: Inverse-Distilled Reward Fine-tuning of Masked Discrete Diffusion Models
  6. 2024Confronting Reward Overoptimization for Diffusion Models: A Perspective of Inductive and Primacy Biases
  7. 2024Avoiding mode collapse in diffusion models fine-tuned with reinforcement learning
  8. 2024Stochastic Control for Fine-tuning Diffusion Models: Optimality, Regularity, and Convergence