mlsanityData to 7 Oct 2026

Learning To Sample From Diffusion Models Via Inverse Reinforcement Learning

arXiv 2602.08689 ↗First posted 9 Feb 2026cs.LGcited by 0

On the map: Diffusion models → Diffusion and flow policies for robotics

also in 1 more
Abstract (arXiv)

Diffusion models generate samples through an iterative denoising process guided by a pretrained neural network. Once the denoiser is fixed, the sampling algorithm itself (noise schedules, guidance scales, stochasticity profiles) still requires careful tuning, a process typically carried out through costly empirical grid search. In this work, we introduce an inverse reinforcement learning framework for learning sampling strategies without retraining the denoiser. We formulate the diffusion sampling procedure as a discrete-time finite-horizon Markov Decision Process, where actions correspond to optional modifications of the sampling dynamics. To optimize action scheduling, we avoid defining an explicit reward function and instead directly match the target behavior expected from the sampler using policy gradient techniques. We provide experimental evidence that this approach matches fine-tuned samplers and comes at a modest cost compared to grid search: on ImageNet-64, a single training run replaces exhaustive search at up to 9x lower cost, with only 16% overhead at inference.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Placed in Diffusion and flow policies for robotics, one of 14 directions in Diffusion models. Builds on 40 of its 47 references that are in the map. First posted on arXiv 9 Feb 2026.

Builds on

40 of 47 references.

Closest ancestors · 5

Same direction, newest first

  1. 2024Avoiding mode collapse in diffusion models fine-tuned with reinforcement learning10 Oct
  2. 2024Adjoint Matching: Fine-tuning Flow and Diffusion Generative Models with Memoryless Stochastic Optimal Control13 Sep
  3. 2024Maximum Entropy Inverse Reinforcement Learning of Diffusion Models with Energy-Based Models30 Jun
  4. 2024Fine-Tuning of Continuous-Time Diffusion Models as Entropy-Regularized Control23 Feb
  5. 2023Training Diffusion Models with Reinforcement Learning22 May

Other references · 35

  1. 2020Denoising Diffusion Probabilistic Modelscited by 20,851
  2. 2017Proximal Policy Optimization Algorithmscited by 14,441
  3. 2020Denoising Diffusion Implicit Modelscited by 9,655
  4. 2020Score-Based Generative Modeling through Stochastic Differential Equationscited by 8,964
  5. 2021Diffusion Models Beat GANs on Image Synthesiscited by 8,185
  6. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 7,916
  7. 2022Photorealistic Text-to-Image Diffusion Models with Deep Language Understandingcited by 6,322
  8. 2018A Style-Based Generator Architecture for Generative Adversarial Networkscited by 5,986
  9. 2022Classifier-Free Diffusion Guidancecited by 5,717
  10. 2022Generative Adversarial Networkscited by 5,603
  11. 2022Elucidating the Design Space of Diffusion-Based Generative Modelscited by 2,994
  12. 2022DPM-Solver: A Fast ODE Solver for Diffusion Probabilistic Model Sampling in Around 10 Stepscited by 1,961
23 more
  1. 2016Generative Adversarial Imitation Learningcited by 1,576
  2. 2019Improved Precision and Recall Metric for Assessing Generative Modelscited by 925
  3. 2023A General Theoretical Paradigm to Understand Learning from Human Preferencescited by 906
  4. 2016f-GAN: Training Generative Neural Samplers using Variational Divergence Minimizationcited by 504
  5. 2021Gotta Go Fast When Generating Data with Score-Based Modelscited by 218
  6. 2019A Divergence Minimization Perspective on Imitation Learning Methodscited by 188
  7. 2022Convergence for score-based generative modeling with polynomial complexitycited by 176
  8. 2023Beyond Reverse KL: Generalizing Direct Preference Optimization with Diverse Divergence Constraintscited by 174
  9. 2016A Connection between Generative Adversarial Networks, Inverse Reinforcement Learning, and Energy-Based Modelscited by 167
  10. 2024Dynamical Regimes of Diffusion Modelscited by 133
  11. 2017Stabilizing Training of Generative Adversarial Networks through Regularizationcited by 118
  12. 2022Refining Generative Process with Discriminator Guidance in Score-based Diffusion Modelscited by 98
  13. 2023Restart Sampling for Improving Generative Processescited by 81
  14. 2020f-IRL: Inverse Reinforcement Learning via State Marginal Matchingcited by 67
  15. 2018Discriminator Rejection Samplingcited by 48
  16. 2024What does guidance do? A fine-grained analysis in a simple settingcited by 45
  17. 2019Bridging the Gap Between f-GANs and Wasserstein GANscited by 23
  18. 2024VersaT2I: Improving Text-to-Image Models with Versatile Rewardcited by 17
  19. 2023f-Policy Gradients: A General Framework for Goal Conditioned RL using f-Divergencescited by 14
  20. 2023TopP&R: Robust Support Estimation Approach for Evaluating Fidelity and Diversity in Generative Modelscited by 13
  21. 2025Provable Maximum Entropy Manifold Exploration via Diffusion Modelscited by 12
  22. 2025Adaptive Diffusion Guidance via Stochastic Optimal Controlcited by 9
  23. 2025Improving Discriminator Guidance in Diffusion Modelscited by 2

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2023Towards Controllable Diffusion Models via Reward-Guided Exploration
  2. 2026Hierarchical Variational Policies for Reward-Guided Diffusion
  3. 2026Diffusion Controller: Framework, Algorithms and Parameterization
  4. 2026Critic-Guided Reinforcement Unlearning in Text-to-Image Diffusion
  5. 2026PAST: Prompt-Adaptive Sampling Termination for Efficient Diffusion Model
  6. 2025Understanding Sampler Stochasticity in Training Diffusion Models for RLHF
  7. 2025Prior-Guided Diffusion Planning for Offline Reinforcement Learning
  8. 2024Solving General Noisy Inverse Problem via Posterior Sampling: A Policy Gradient Viewpoint