mlsanityData to 14 Oct 2026

Noisy-Space Policy Gradient for Diffusion Policies in Offline Reinforcement Learning

arXiv 2609.06882 ↗First posted 7 Sep 2026cs.LG cs.AI cs.ROcited by 1

On the map: Diffusion models → Diffusion and flow policies for robotics

also in 1 more
Abstract (arXiv)

Diffusion policies offer a powerful and expressive parameterization for continuous control. Yet, their integration with reinforcement learning remains conceptually and algorithmically challenging. In this work, we address this gap by introducing a noisy-space action-value (Q-)function that assigns values to diffusion latents through the distribution of executed actions induced by the denoising process. We show that this construction admits a precise semantic interpretation and derive a noisy-space policy gradient (NSPG) that optimizes noisy latents using only clean action-space value estimates. Building on this result, we formulate a KL-regularized policy improvement over noisy latents and show that the resulting objective admits a diffusion-compatible regression form, avoiding backpropagation through the denoising process. Empirical results on state-based D4RL benchmarks and vision-based OGBench tasks demonstrate that the proposed noisy-space objective provides a principled and effective basis for training diffusion policies in offline reinforcement learning. Project webpage: https://mahmoud-selim.github.io/NSPG/

Key facts. Cited by 1 paper in the map (data through 14 Oct 2026). Placed in Diffusion and flow policies for robotics, one of 14 directions in Diffusion models. Builds on 35 of its 40 references that are in the map. First posted on arXiv 7 Sep 2026.

Builds on

35 of 40 references.

Closest ancestors · 11

Same direction, newest first

  1. 2025ReinFlow: Fine-tuning Flow Matching Policy with Online Reinforcement Learning28 May
  2. 2025Behavior-Regularized Diffusion Policy Optimization for Offline Reinforcement Learning7 Feb
  3. 2025Flow Q-Learning4 Feb
  4. 2024Diffusion Actor-Critic: Formulating Constrained Policy Iteration as Diffusion Noise Regression for Offline Reinforcement Learning31 May
  5. 2024Diffusion Policies creating a Trust Region for Offline Reinforcement Learning30 May
  6. 2024Entropy-regularized Diffusion Policy with Q-Ensembles for Offline Reinforcement Learning6 Feb
  7. 2023DiffCPS: Diffusion Model based Constrained Policy Search for Offline Reinforcement Learning9 Oct
  8. 2023Consistency Models as a Rich and Efficient Policy Class for Reinforcement Learning29 Sep
  9. 2023Diffusion Policies for Out-of-Distribution Generalization in Offline Reinforcement Learning10 Jul
  10. 2023IDQL: Implicit Q-Learning as an Actor-Critic Method with Diffusion Policies20 Apr
  11. 2022Diffusion Policies as an Expressive Policy Class for Offline Reinforcement Learning12 Aug

Other references · 24

  1. 2020Denoising Diffusion Probabilistic Modelscited by 21,076
  2. 2017Proximal Policy Optimization Algorithmscited by 15,215
  3. 2015Deep Unsupervised Learning using Nonequilibrium Thermodynamicscited by 7,058
  4. 2015Trust Region Policy Optimizationcited by 2,580
  5. 2020Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problemscited by 1,920
  6. 2020Conservative Q-Learning for Offline Reinforcement Learningcited by 1,809
  7. 2021Offline Reinforcement Learning with Implicit Q-Learningcited by 1,293
  8. 2018Off-Policy Deep Reinforcement Learning without Explorationcited by 1,241
  9. 2022Planning with Diffusion for Flexible Behavior Synthesiscited by 1,061
  10. 2021A Minimalist Approach to Offline Reinforcement Learningcited by 899
  11. 2019Stabilizing Off-Policy Q-Learning via Bootstrapping Error Reductioncited by 840
  12. 2019Behavior Regularized Offline Reinforcement Learningcited by 662
12 more
  1. 2020MOReL : Model-Based Offline Reinforcement Learningcited by 590
  2. 2021Uncertainty-Based Offline Reinforcement Learning with Diversified Q-Ensemblecited by 302
  3. 2022Offline Reinforcement Learning via High-Fidelity Generative Behavior Modelingcited by 182
  4. 2021Offline RL Without Off-Policy Evaluationcited by 151
  5. 2023Revisiting the Minimalist Approach to Offline Reinforcement Learningcited by 126
  6. 2021OptiDICE: Offline Policy Optimization via Stationary Distribution Correction Estimationcited by 121
  7. 2023Extreme Q-Learning: MaxEnt RL without Entropycited by 116
  8. 2023Offline RL with No OOD Actions: In-Sample Learning via Implicit Value Regularizationcited by 116
  9. 2022Why So Pessimistic? Estimating Uncertainties for Offline RL through Ensembles, and Why Their Independence Matterscited by 76
  10. 2023Anti-Exploration by Random Network Distillationcited by 50
  11. 2023Dual RL: Unification and New Methods for Reinforcement and Imitation Learningcited by 48
  12. 2023Behavior Proximal Policy Optimizationcited by 43

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 1.

1 paper.

  1. 2026Beyond Policy Support: Interaction Constrained Offline Reinforcement Learning for Autonomous Drivingcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024Preferred-Action-Optimized Diffusion Policies for Offline Reinforcement Learning
  2. 2023Score Regularized Policy Optimization through Diffusion Behavior
  3. 2025Prior-Guided Diffusion Planning for Offline Reinforcement Learning
  4. 2025Offline Reinforcement Learning with Penalized Action Noise Injection
  5. 2026Diffusion Policy through Conditional Proximal Policy Optimization
  6. 2026Reinforcement Learning from Denoising Feedback
  7. 2025Diffusion Policies with Value-Conditional Optimization for Offline Reinforcement Learning
  8. 2026Designing Reinforcement Learning for Diffusion Models: A Unified Path-Space View