# Is RL fine-tuning harder than regression? A PDE learning approach for diffusion models

arXiv: https://arxiv.org/abs/2509.02528 · HTML: https://mlsanity.com/paper/2509.02528/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Placed in Diffusion and flow policies for robotics, one of 14 directions in Diffusion models.
- Builds on 20 of its 65 references that are in the map.
- First posted on arXiv 2 Sep 2025.

## Placed in

- Diffusion models / [Diffusion and flow policies for robotics](https://mlsanity.com/diffusion/diffusion-and-flow-policies-for-robotics/)

## Builds on (most cited first, 20 of 20)

- [DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models](https://mlsanity.com/paper/2402.03300/) (2024)
- [Score-Based Generative Modeling through Stochastic Differential Equations](https://mlsanity.com/paper/2011.13456/) (2020)
- [Fine-Tuning Language Models from Human Preferences](https://mlsanity.com/paper/1909.08593/) (2019)
- [Training Diffusion Models with Reinforcement Learning](https://mlsanity.com/paper/2305.13301/) (2023)
- [Fine-Tuning of Continuous-Time Diffusion Models as Entropy-Regularized Control](https://mlsanity.com/paper/2402.15194/) (2024)
- [Understanding Reinforcement Learning-Based Fine-Tuning of Diffusion Models: A Tutorial and Review](https://mlsanity.com/paper/2407.13734/) (2024)
- [Reproducibility of Benchmarked Deep Reinforcement Learning Tasks for Continuous Control](https://mlsanity.com/paper/1708.04133/) (2017)
- [Optimizing DDPM Sampling with Shortcut Fine-Tuning](https://mlsanity.com/paper/2301.13362/) (2023)
- [Feedback Efficient Online Fine-Tuning of Diffusion Models](https://mlsanity.com/paper/2402.16359/) (2024)
- [Fine-tuning of diffusion models via stochastic control: entropy regularization and beyond](https://mlsanity.com/paper/2403.06279/) (2024)
- [Machine Learning For Elliptic PDEs: Fast Rate Generalization Bound, Neural Scaling Law and Minimax Optimality](https://mlsanity.com/paper/2110.06897/) (2021)
- [Score as Action: Fine-Tuning Diffusion Generative Models by Continuous-time Reinforcement Learning](https://mlsanity.com/paper/2502.01819/) (2025)
- [Accelerating RL for LLM Reasoning with Optimal Advantage Regression](https://mlsanity.com/paper/2505.20686/) (2025)
- [Reward-Directed Score-Based Diffusion Models via q-Learning](https://mlsanity.com/paper/2409.04832/) (2024)
- [Stochastic Control for Fine-tuning Diffusion Models: Optimality, Regularity, and Convergence](https://mlsanity.com/paper/2412.18164/) (2024)
- [Trajectory Bellman Residual Minimization: A Simple Value-Based Method for LLM Reasoning](https://mlsanity.com/paper/2505.15311/) (2025)
- [On Bellman equations for continuous-time policy evaluation I: discretization and approximation](https://mlsanity.com/paper/2407.05966/) (2024)
- [Deep neural network approximation for high-dimensional parabolic Hamilton-Jacobi-Bellman equations](https://mlsanity.com/paper/2103.05744/) (2021)
- [Statistical guarantees for continuous-time policy evaluation: blessing of ellipticity and new tradeoffs](https://mlsanity.com/paper/2502.04297/) (2025)
- [Surprising Negative Results for Generative Adversarial Tree Search](https://mlsanity.com/paper/1806.05780/) (2018)

## Built on by (most cited first, 3 of 3)

- [Inference-Time Alignment for Diffusion Models via Variationally Stable Doob's Matching](https://mlsanity.com/paper/2601.06514/) (2026)
- [Continuous-time reinforcement learning: ellipticity enables model-free value function approximation](https://mlsanity.com/paper/2602.06930/) (2026)
- [Schrödinger--Föllmer Actor--Critic: Diffusion Policy Improvement with Finite-Sample Analysis](https://mlsanity.com/paper/2609.33239/) (2026)

## Abstract (arXiv)

We study the problem of learning the optimal control policy for fine-tuning a given diffusion process, using general value function approximation. We develop a new class of algorithms by solving a variational inequality problem based on the Hamilton-Jacobi-Bellman (HJB) equations. We prove sharp statistical rates for the learned value function and control policy, depending on the complexity and approximation errors of the function class. In contrast to generic reinforcement learning problems, our approach shows that fine-tuning can be achieved via supervised regression, with faster statistical rate guarantees.