# Fine-tuning of diffusion models via stochastic control: entropy regularization and beyond

arXiv: https://arxiv.org/abs/2403.06279 · HTML: https://mlsanity.com/paper/2403.06279/

## Key facts

- Cited by 42 papers in the map (data through 14 Oct 2026).
- Placed in Diffusion foundations and sampling, one of 14 directions in Diffusion models.
- Builds on 27 of its 55 references that are in the map.
- First posted on arXiv 10 Mar 2024.

## Placed in

- Diffusion models / [Diffusion foundations and sampling](https://mlsanity.com/diffusion/diffusion-foundations-and-sampling/)

## Builds on (most cited first, 27 of 27)

- [Denoising Diffusion Probabilistic Models](https://mlsanity.com/paper/2006.11239/) (2020)
- [High-Resolution Image Synthesis with Latent Diffusion Models](https://mlsanity.com/paper/2112.10752/) (2021)
- [Score-Based Generative Modeling through Stochastic Differential Equations](https://mlsanity.com/paper/2011.13456/) (2020)
- [Diffusion Models Beat GANs on Image Synthesis](https://mlsanity.com/paper/2105.05233/) (2021)
- [Direct Preference Optimization: Your Language Model is Secretly a Reward Model](https://mlsanity.com/paper/2305.18290/) (2023)
- [Deep Unsupervised Learning using Nonequilibrium Thermodynamics](https://mlsanity.com/paper/1503.03585/) (2015)
- [Hierarchical Text-Conditional Image Generation with CLIP Latents](https://mlsanity.com/paper/2204.06125/) (2022)
- [Classifier-Free Diffusion Guidance](https://mlsanity.com/paper/2207.12598/) (2022)
- [Neural Ordinary Differential Equations](https://mlsanity.com/paper/1806.07366/) (2018)
- [Convex optimization](https://mlsanity.com/paper/2106.01946/) (2021)
- [DiffWave: A Versatile Diffusion Model for Audio Synthesis](https://mlsanity.com/paper/2009.09761/) (2020)
- [Training Diffusion Models with Reinforcement Learning](https://mlsanity.com/paper/2305.13301/) (2023)
- [Diffusion Model Alignment Using Direct Preference Optimization](https://mlsanity.com/paper/2311.12908/) (2023)
- [Diffusion Schr"odinger Bridge with Applications to Score-Based Generative Modeling](https://mlsanity.com/paper/2106.01357/) (2021)
- [DPOK: Reinforcement Learning for Fine-tuning Text-to-Image Diffusion Models](https://mlsanity.com/paper/2305.16381/) (2023)
- [Sampling is as easy as learning the score: theory for diffusion models with minimal data assumptions](https://mlsanity.com/paper/2209.11215/) (2022)
- [Sliced Score Matching: A Scalable Approach to Density and Score Estimation](https://mlsanity.com/paper/1905.07088/) (2019)
- [Beyond Reverse KL: Generalizing Direct Preference Optimization with Diverse Divergence Constraints](https://mlsanity.com/paper/2309.16240/) (2023)
- [Fine-Tuning of Continuous-Time Diffusion Models as Entropy-Regularized Control](https://mlsanity.com/paper/2402.15194/) (2024)
- [Optimizing DDPM Sampling with Shortcut Fine-Tuning](https://mlsanity.com/paper/2301.13362/) (2023)
- [Efficient and Accurate Gradients for Neural SDEs](https://mlsanity.com/paper/2105.13493/) (2021)
- [Score-based Diffusion Models via Stochastic Differential Equations -- a Technical Tutorial](https://mlsanity.com/paper/2402.07487/) (2024)
- [Diffusion Models as Constrained Samplers for Optimization with Unknown Constraints](https://mlsanity.com/paper/2402.18012/) (2024)
- [Exploratory HJB equations and their convergence](https://mlsanity.com/paper/2109.10269/) (2021)
- [Contractive Diffusion Probabilistic Models](https://mlsanity.com/paper/2401.13115/) (2024)
- [Reward Collapse in Aligning Large Language Models](https://mlsanity.com/paper/2305.17608/) (2023)
- [State-Dependent Temperature Control for Langevin Diffusions](https://mlsanity.com/paper/2011.07456/) (2020)

## Built on by (most cited first, 30 of 42)

- [Adjoint Matching: Fine-tuning Flow and Diffusion Generative Models with Memoryless Stochastic Optimal Control](https://mlsanity.com/paper/2409.08861/) (2024)
- [Understanding Reinforcement Learning-Based Fine-Tuning of Diffusion Models: A Tutorial and Review](https://mlsanity.com/paper/2407.13734/) (2024)
- [Decoding-Time Language Model Alignment with Multiple Objectives](https://mlsanity.com/paper/2406.18853/) (2024)
- [An Overview of Diffusion Models: Applications, Guided Generation, Statistical Rates and Optimization](https://mlsanity.com/paper/2404.07771/) (2024)
- [One-step Diffusion Models with f-Divergence Distribution Matching](https://mlsanity.com/paper/2502.15681/) (2025)
- [Noise Hypernetworks: Amortizing Test-Time Compute in Diffusion Models](https://mlsanity.com/paper/2508.09968/) (2025)
- [Score as Action: Fine-Tuning Diffusion Generative Models by Continuous-time Reinforcement Learning](https://mlsanity.com/paper/2502.01819/) (2025)
- [Preference Tuning with Human Feedback on Language, Speech, and Vision Tasks: A Survey](https://mlsanity.com/paper/2409.11564/) (2024)
- [Constrained Diffusion Models via Dual Training](https://mlsanity.com/paper/2408.15094/) (2024)
- [Reward-Directed Score-Based Diffusion Models via q-Learning](https://mlsanity.com/paper/2409.04832/) (2024)
- [Scores as Actions: a framework of fine-tuning diffusion models by continuous-time reinforcement learning](https://mlsanity.com/paper/2409.08400/) (2024)
- [Stochastic Control for Fine-tuning Diffusion Models: Optimality, Regularity, and Convergence](https://mlsanity.com/paper/2412.18164/) (2024)
- [Flow Density Control: Generative Optimization Beyond Entropy-Regularized Fine-Tuning](https://mlsanity.com/paper/2511.22640/) (2025)
- [Psi-Sampler: Initial Particle Sampling for SMC-Based Inference-Time Reward Alignment in Score Models](https://mlsanity.com/paper/2506.01320/) (2025)
- [Provable Maximum Entropy Manifold Exploration via Diffusion Models](https://mlsanity.com/paper/2506.15385/) (2025)
- [TR2-D2: Tree Search Guided Trajectory-Aware Fine-Tuning for Discrete Diffusion](https://mlsanity.com/paper/2509.25171/) (2025)
- [Source-Guided Flow Matching](https://mlsanity.com/paper/2508.14807/) (2025)
- [Conditional Diffusion Guidance under Hard Constraint: A Stochastic Analysis Approach](https://mlsanity.com/paper/2602.05533/) (2026)
- [Efficient Tail-Aware Generative Optimization via Flow Model Fine-Tuning](https://mlsanity.com/paper/2602.16796/) (2026)
- [OPD+: Rethinking the Advantage Design for On-Policy Distillation](https://mlsanity.com/paper/2606.01039/) (2026)
- [Understanding Sampler Stochasticity in Training Diffusion Models for RLHF](https://mlsanity.com/paper/2510.10767/) (2025)
- [Calibrating Generative Models to Distributional Constraints](https://mlsanity.com/paper/2510.10020/) (2025)
- [Adjoint Matching through the Lens of the Stochastic Maximum Principle in Optimal Control](https://mlsanity.com/paper/2604.08580/) (2026)
- [Direct Distributional Optimization for Provable Alignment of Diffusion Models](https://mlsanity.com/paper/2502.02954/) (2025)
- [Inference-Time Alignment for Diffusion Models via Variationally Stable Doob's Matching](https://mlsanity.com/paper/2601.06514/) (2026)
- [Improved techniques for fine-tuning flow models via adjoint matching: a deterministic control pipeline](https://mlsanity.com/paper/2605.06583/) (2026)
- [Maximum Entropy Inverse Reinforcement Learning of Diffusion Models with Energy-Based Models](https://mlsanity.com/paper/2407.00626/) (2024)
- [A First-order Generative Bilevel Optimization Framework for Diffusion Models](https://mlsanity.com/paper/2502.08808/) (2025)
- [Iterative Tilting for Diffusion Fine-Tuning](https://mlsanity.com/paper/2512.03234/) (2025)
- [f-Trajectory Balance: A Loss Family for Tuning GFlowNets, Generative Models, and LLMs with Off- and On-Policy Data](https://mlsanity.com/paper/2605.15417/) (2026)

## Abstract (arXiv)

This paper aims to develop and provide a rigorous treatment to the problem of entropy regularized fine-tuning in the context of continuous-time diffusion models, which was recently proposed by Uehara et al. (arXiv:2402.15194, 2024). The idea is to use stochastic control for sample generation, where the entropy regularizer is introduced to mitigate reward collapse. We also show how the analysis can be extended to fine-tuning with a general f-divergence regularizer. Numerical experiments on large-scale text-to-image models--Stable Diffusion v1.5 are conducted to validate our approach.