mlsanityData to 7 Oct 2026

DiPOD: Diffusion Policy Optimization without Drifting Apart

arXiv 2606.13795 ↗First posted 11 Jun 2026cs.LGcited by 0

On the map: Diffusion models → Diffusion and flow policies for robotics

Abstract (arXiv)

RL post-training has become increasingly pivotal for improving diffusion policies, but existing diffusion policy-gradient methods are often unstable and cannot achieve reliable policy improvement. We identify the cause as the double-drift phenomenon: optimizing a variational surrogate can let the ELBO separate from the true log-likelihood, which then makes the resulting proxy policy gradient misaligned with the true policy gradient of expected return. We propose DiPOD, a diffusion policy optimization framework that maintains tight-bound behavior throughout training by interleaving self-distillation with policy-improving gradient updates. This leads to a simple and practical algorithm: augmenting each diffusion policy-gradient update with an on-policy ELBO regularizer. Across diffusion language model post-training and continuous-control diffusion policies, DiPOD substantially stabilizes training and reaches higher rewards than previous methods.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Placed in Diffusion and flow policies for robotics, one of 14 directions in Diffusion models. Builds on 44 of its 49 references that are in the map. First posted on arXiv 11 Jun 2026.

Builds on

44 of 49 references.

Closest ancestors · 9

Same direction, newest first

  1. 2026Q-learning with Adjoint Matching20 Jan
  2. 2025Flow Matching Policy Gradients28 Jul
  3. 2025Steering Your Diffusion Policy with Latent Space Reinforcement Learning18 Jun
  4. 2024Adjoint Matching: Fine-tuning Flow and Diffusion Generative Models with Memoryless Stochastic Optimal Control13 Sep
  5. 2024Diffusion Policy Policy Optimization1 Sep
  6. 2024Diffusion Actor-Critic: Formulating Constrained Policy Iteration as Diffusion Noise Regression for Offline Reinforcement Learning31 May
  7. 2024Diffusion-based Reinforcement Learning via Q-weighted Variational Policy Optimization25 May
  8. 2023Training Diffusion Models with Reinforcement Learning22 May
  9. 2023IDQL: Implicit Q-Learning as an Actor-Critic Method with Diffusion Policies20 Apr

Other references · 35

  1. 2020Denoising Diffusion Probabilistic Modelscited by 20,851
  2. 2017Proximal Policy Optimization Algorithmscited by 14,441
  3. 2021Training Verifiers to Solve Math Word Problemscited by 9,743
  4. 2020Denoising Diffusion Implicit Modelscited by 9,655
  5. 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 8,851
  6. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 7,916
  7. 2015Deep Unsupervised Learning using Nonequilibrium Thermodynamicscited by 7,005
  8. 2019Generative Modeling by Estimating Gradients of the Data Distributioncited by 4,067
  9. 2023Let's Verify Step by Stepcited by 3,598
  10. 2024π0: A Vision-Language-Action Flow Model for General Robot Controlcited by 2,297
  11. 2022Video Diffusion Modelscited by 2,280
  12. 2015High-Dimensional Continuous Control Using Generalized Advantage Estimationcited by 1,916
23 more
  1. 2022Imagen Video: High Definition Video Generation with Diffusion Modelscited by 1,765
  2. 2025GR00T N1: An Open Foundation Model for Generalist Humanoid Robotscited by 1,167
  3. 2025Large Language Diffusion Modelscited by 872
  4. 2021Learning to Walk in Minutes Using Massively Parallel Deep Reinforcement Learningcited by 395
  5. 2023Understanding Diffusion Objectives as the ELBO with Simple Data Augmentationcited by 293
  6. 2024Guiding a Diffusion Model with a Bad Version of Itselfcited by 286
  7. 2025MMaDA: Multimodal Large Diffusion Language Modelscited by 241
  8. 2021Robust Motion In-betweeningcited by 223
  9. 2023Reduce, Reuse, Recycle: Compositional Generation with Energy-Based Diffusion Models and MCMCcited by 216
  10. 2025d1: Scaling Reasoning in Diffusion Large Language Models via Reinforcement Learningcited by 178
  11. 2025Seed Diffusion: A Large-Scale Diffusion Language Model with High-Speed Inferencecited by 173
  12. 2020Parrot: Data-Driven Behavioral Priors for Reinforcement Learningcited by 136
  13. 2026DFlash: Block Diffusion for Flash Speculative Decodingcited by 120
  14. 2025Reinforcement Learning with Action Chunkingcited by 97
  15. 2025Dream-Coder 7B: An Open Diffusion Language Model for Codecited by 81
  16. 2024From Imitation to Refinement -- Residual RL for Precise Assemblycited by 80
  17. 2024Policy Agnostic RL: Offline RL and Online RL Fine-Tuning of Any Class and Backbonecited by 71
  18. 2025wd1: Weighted Policy Optimization for Reasoning in Diffusion Language Modelscited by 67
  19. 2024Policy Decorator: Model-Agnostic Online Refinement for Large Policy Modelcited by 60
  20. 2025SPG: Sandwiched Policy Gradient for Masked Diffusion Language Modelscited by 38
  21. 2025EXPO: Stable Reinforcement Learning with Expressive Policiescited by 30
  22. 2026Flow Policy Gradients for Robot Controlcited by 14
  23. 2025Diffusion Language Models are Provably Optimal Parallel Samplerscited by 5

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2025Dichotomous Diffusion Policy Optimization
  2. 2026iADD: Improving Alignment and Diversity in Diffusion Policy Optimization
  3. 2026On-Policy Self-Distillation in Diffusion Models
  4. 2026Diffusion Policy through Conditional Proximal Policy Optimization
  5. 2026Diffusion Controller: Framework, Algorithms and Parameterization
  6. 2026Explicit Critic Guidance for Aligning Diffusion Models
  7. 2023Score Regularized Policy Optimization through Diffusion Behavior
  8. 2025MDPO: Overcoming the Training-Inference Divide of Masked Diffusion Language Models