mlsanityData to 7 Oct 2026

Fine-tuning of diffusion models via stochastic control: entropy regularization and beyond

arXiv 2403.06279 ↗First posted 10 Mar 2024math.OC cs.LGcited by 42

On the map: Diffusion models → Diffusion foundations and sampling

Abstract (arXiv)

This paper aims to develop and provide a rigorous treatment to the problem of entropy regularized fine-tuning in the context of continuous-time diffusion models, which was recently proposed by Uehara et al. (arXiv:2402.15194, 2024). The idea is to use stochastic control for sample generation, where the entropy regularizer is introduced to mitigate reward collapse. We also show how the analysis can be extended to fine-tuning with a general f-divergence regularizer. Numerical experiments on large-scale text-to-image models--Stable Diffusion v1.5 are conducted to validate our approach.

Key facts. Cited by 42 papers in the map (data through 7 Oct 2026). Placed in Diffusion foundations and sampling, one of 14 directions in Diffusion models. Builds on 27 of its 55 references that are in the map. First posted on arXiv 10 Mar 2024.

Builds on

27 of 55 references.

Closest ancestors · 11

Same direction, newest first

  1. 2024Diffusion Models as Constrained Samplers for Optimization with Unknown Constraints28 Feb
  2. 2024Score-based Diffusion Models via Stochastic Differential Equations -- a Technical Tutorial12 Feb
  3. 2024Contractive Diffusion Probabilistic Models23 Jan
  4. 2023Optimizing DDPM Sampling with Shortcut Fine-Tuning31 Jan
  5. 2022Sampling is as easy as learning the score: theory for diffusion models with minimal data assumptions22 Sep
  6. 2022Classifier-Free Diffusion Guidance26 Jul
  7. 2021High-Resolution Image Synthesis with Latent Diffusion Models20 Dec
  8. 2021Diffusion Schr"odinger Bridge with Applications to Score-Based Generative Modeling1 Jun
  9. 2021Diffusion Models Beat GANs on Image Synthesis11 May
  10. 2020Score-Based Generative Modeling through Stochastic Differential Equations26 Nov
  11. 2020Denoising Diffusion Probabilistic Models19 Jun

Other references · 16

  1. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 7,916
  2. 2015Deep Unsupervised Learning using Nonequilibrium Thermodynamicscited by 7,005
  3. 2022Hierarchical Text-Conditional Image Generation with CLIP Latentscited by 6,539
  4. 2018Neural Ordinary Differential Equationscited by 3,733
  5. 2021Convex optimizationcited by 2,479
  6. 2020DiffWave: A Versatile Diffusion Model for Audio Synthesiscited by 1,339
  7. 2023Training Diffusion Models with Reinforcement Learningcited by 939
  8. 2023Diffusion Model Alignment Using Direct Preference Optimizationcited by 767
  9. 2023DPOK: Reinforcement Learning for Fine-tuning Text-to-Image Diffusion Modelscited by 431
  10. 2019Sliced Score Matching: A Scalable Approach to Density and Score Estimationcited by 381
  11. 2023Beyond Reverse KL: Generalizing Direct Preference Optimization with Diverse Divergence Constraintscited by 174
  12. 2024Fine-Tuning of Continuous-Time Diffusion Models as Entropy-Regularized Controlcited by 136
4 more
  1. 2021Efficient and Accurate Gradients for Neural SDEscited by 68
  2. 2021Exploratory HJB equations and their convergencecited by 28
  3. 2023Reward Collapse in Aligning Large Language Modelscited by 23
  4. 2020State-Dependent Temperature Control for Langevin Diffusionscited by 17

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 10; 2025: 17; 2026: 15.

42 papers.

Sort:

By area:

  1. 2024Adjoint Matching: Fine-tuning Flow and Diffusion Generative Models with Memoryless Stochastic Optimal Controlcited by 222
  2. 2024Understanding Reinforcement Learning-Based Fine-Tuning of Diffusion Models: A Tutorial and Reviewcited by 92
  3. 2024Decoding-Time Language Model Alignment with Multiple Objectivescited by 90
  4. 2024An Overview of Diffusion Models: Applications, Guided Generation, Statistical Rates and Optimizationcited by 72
  5. 2025One-step Diffusion Models with f-Divergence Distribution Matchingcited by 43
  6. 2025Noise Hypernetworks: Amortizing Test-Time Compute in Diffusion Modelscited by 35
  7. 2025Score as Action: Fine-Tuning Diffusion Generative Models by Continuous-time Reinforcement Learningcited by 33
  8. 2024Preference Tuning with Human Feedback on Language, Speech, and Vision Tasks: A Surveycited by 23
  9. 2024Constrained Diffusion Models via Dual Trainingcited by 19
  10. 2024Scores as Actions: a framework of fine-tuning diffusion models by continuous-time reinforcement learningcited by 19
  11. 2024Reward-Directed Score-Based Diffusion Models via q-Learningcited by 18
  12. 2024Stochastic Control for Fine-tuning Diffusion Models: Optimality, Regularity, and Convergencecited by 18
30 more
  1. 2025Psi-Sampler: Initial Particle Sampling for SMC-Based Inference-Time Reward Alignment in Score Modelscited by 15
  2. 2025Flow Density Control: Generative Optimization Beyond Entropy-Regularized Fine-Tuningcited by 15
  3. 2025TR2-D2: Tree Search Guided Trajectory-Aware Fine-Tuning for Discrete Diffusioncited by 13
  4. 2025Provable Maximum Entropy Manifold Exploration via Diffusion Modelscited by 12
  5. 2025Source-Guided Flow Matchingcited by 11
  6. 2026Conditional Diffusion Guidance under Hard Constraint: A Stochastic Analysis Approachcited by 10
  7. 2025Understanding Sampler Stochasticity in Training Diffusion Models for RLHFcited by 7
  8. 2026OPD+: Rethinking the Advantage Design for On-Policy Distillationcited by 7
  9. 2026Efficient Tail-Aware Generative Optimization via Flow Model Fine-Tuningcited by 6
  10. 2025Calibrating Generative Models to Distributional Constraintscited by 5
  11. 2024Maximum Entropy Inverse Reinforcement Learning of Diffusion Models with Energy-Based Modelscited by 4
  12. 2025Direct Distributional Optimization for Provable Alignment of Diffusion Modelscited by 4
  13. 2025A First-order Generative Bilevel Optimization Framework for Diffusion Modelscited by 4
  14. 2025Iterative Tilting for Diffusion Fine-Tuningcited by 4
  15. 2026Adjoint Matching through the Lens of the Stochastic Maximum Principle in Optimal Controlcited by 4
  16. 2025Is RL fine-tuning harder than regression? A PDE learning approach for diffusion modelscited by 3
  17. 2026Improved techniques for fine-tuning flow models via adjoint matching: a deterministic control pipelinecited by 3
  18. 2026f-Trajectory Balance: A Loss Family for Tuning GFlowNets, Generative Models, and LLMs with Off- and On-Policy Datacited by 3
  19. 2025A Unified Framework for Diffusion Model Unlearning with f-Divergencecited by 2
  20. 2025Schr"odinger bridge for generative AI: Soft-constrained formulation and convergence analysiscited by 2
  21. 2026Inference-Time Alignment for Diffusion Models via Variationally Stable Doob's Matchingcited by 2
  22. 2026A Unified Density Operator View of Flow Control and Mergingcited by 2
  23. 2026Constrained Flow Optimization via Sequential Fine Tuning for Molecular Designcited by 2
  24. 2026Q-VGM: Q-Guided Value-Gradient Matching for Offline-to-Online RL of Flow-Matching VLA Policiescited by 2
  25. 2025MIRO: MultI-Reward cOnditioned pretraining improves T2I quality and efficiencycited by 1
  26. 2026Diffusion Controller: Framework, Algorithms and Parameterizationcited by 1
  27. 2026A Stabilized Path-Space Approach to Diffusion-Based Posterior Samplingcited by 1
  28. 2026Hierarchical Variational Policies for Reward-Guided Diffusioncited by 0
  29. 2026Sobolev Regularized Score Difference Estimation in Diffusion Modelscited by 0
  30. 2026Fenchel Tilting: Weighted Correction for Efficient Finetuning of Generative Modelscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026A unified perspective on fine-tuning and sampling with diffusion and flow models
  2. 2026An Analysis of Regularization and Fokker-Planck Residuals in Diffusion Models for Image Generation
  3. 2025Iterative Importance Fine-tuning of Diffusion Models
  4. 2022An optimal control perspective on diffusion-based generative modeling
  5. 2025Fine-Tuning Diffusion Models via Intermediate Distribution Shaping
  6. 2023Towards Controllable Diffusion Models via Reward-Guided Exploration
  7. 2024Implicit Diffusion: Efficient Optimization through Stochastic Sampling
  8. 2025Test-time Alignment of Diffusion Models without Reward Over-optimization