mlsanityData to 14 Oct 2026

Is RL fine-tuning harder than regression? A PDE learning approach for diffusion models

arXiv 2509.02528 ↗First posted 2 Sep 2025cs.LG math.OC math.PRcited by 3

On the map: Diffusion models → Diffusion and flow policies for robotics

Abstract (arXiv)

We study the problem of learning the optimal control policy for fine-tuning a given diffusion process, using general value function approximation. We develop a new class of algorithms by solving a variational inequality problem based on the Hamilton-Jacobi-Bellman (HJB) equations. We prove sharp statistical rates for the learned value function and control policy, depending on the complexity and approximation errors of the function class. In contrast to generic reinforcement learning problems, our approach shows that fine-tuning can be achieved via supervised regression, with faster statistical rate guarantees.

Key facts. Cited by 3 papers in the map (data through 14 Oct 2026). Placed in Diffusion and flow policies for robotics, one of 14 directions in Diffusion models. Builds on 20 of its 65 references that are in the map. First posted on arXiv 2 Sep 2025.

Builds on

20 of 65 references.

Closest ancestors · 7

Same direction, newest first

  1. 2025Score as Action: Fine-Tuning Diffusion Generative Models by Continuous-time Reinforcement Learning3 Feb
  2. 2024Stochastic Control for Fine-tuning Diffusion Models: Optimality, Regularity, and Convergence24 Dec
  3. 2024Reward-Directed Score-Based Diffusion Models via q-Learning7 Sep
  4. 2024Understanding Reinforcement Learning-Based Fine-Tuning of Diffusion Models: A Tutorial and Review18 Jul
  5. 2024Feedback Efficient Online Fine-Tuning of Diffusion Models26 Feb
  6. 2024Fine-Tuning of Continuous-Time Diffusion Models as Entropy-Regularized Control23 Feb
  7. 2023Training Diffusion Models with Reinforcement Learning22 May

Other references · 13

  1. 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 9,379
  2. 2020Score-Based Generative Modeling through Stochastic Differential Equationscited by 9,122
  3. 2019Fine-Tuning Language Models from Human Preferencescited by 2,150
  4. 2017Reproducibility of Benchmarked Deep Reinforcement Learning Tasks for Continuous Controlcited by 93
  5. 2023Optimizing DDPM Sampling with Shortcut Fine-Tuningcited by 81
  6. 2024Fine-tuning of diffusion models via stochastic control: entropy regularization and beyondcited by 42
  7. 2021Machine Learning For Elliptic PDEs: Fast Rate Generalization Bound, Neural Scaling Law and Minimax Optimalitycited by 40
  8. 2025Accelerating RL for LLM Reasoning with Optimal Advantage Regressioncited by 24
  9. 2025Trajectory Bellman Residual Minimization: A Simple Value-Based Method for LLM Reasoningcited by 11
  10. 2024On Bellman equations for continuous-time policy evaluation I: discretization and approximationcited by 7
  11. 2021Deep neural network approximation for high-dimensional parabolic Hamilton-Jacobi-Bellman equationscited by 5
  12. 2025Statistical guarantees for continuous-time policy evaluation: blessing of ellipticity and new tradeoffscited by 4
1 more
  1. 2018Surprising Negative Results for Generative Adversarial Tree Searchcited by 2

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 3.

3 papers.

  1. 2026Inference-Time Alignment for Diffusion Models via Variationally Stable Doob's Matchingcited by 5
  2. 2026Continuous-time reinforcement learning: ellipticity enables model-free value function approximationcited by 0
  3. 2026Schrödinger--Föllmer Actor--Critic: Diffusion Policy Improvement with Finite-Sample Analysiscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025VARD: Efficient and Dense Fine-Tuning for Diffusion Models with Value-based RL
  2. 2024Avoiding mode collapse in diffusion models fine-tuned with reinforcement learning
  3. 2026Diffusion Controller: Framework, Algorithms and Parameterization
  4. 2025Fine-tuning Diffusion Policies with Backpropagation Through Diffusion Timesteps
  5. 2026Learning to Solve Stochastic Controls with Unknown Drifts and Running Rewards: Theory, Algorithms and Convergence
  6. 2026A unified perspective on fine-tuning and sampling with diffusion and flow models
  7. 2024Adding Conditional Control to Diffusion Models with Reinforcement Learning
  8. 2026Reinforcement Learning for Discounted and Ergodic Control of Diffusion Processes