mlsanityData to 7 Oct 2026

Diffusion Policy Policy Optimization

arXiv 2409.00588 ↗First posted 1 Sep 2024cs.RO cs.LGcited by 255

On the map: Diffusion models → Diffusion and flow policies for robotics

Abstract (arXiv)

We introduce Diffusion Policy Policy Optimization, DPPO, an algorithmic framework including best practices for fine-tuning diffusion-based policies (e.g. Diffusion Policy) in continuous control and robot learning tasks using the policy gradient (PG) method from reinforcement learning (RL). PG methods are ubiquitous in training RL policies with other policy parameterizations; nevertheless, they had been conjectured to be less efficient for diffusion-based policies. Surprisingly, we show that DPPO achieves the strongest overall performance and efficiency for fine-tuning in common benchmarks compared to other RL methods for diffusion-based policies and also compared to PG fine-tuning of other policy parameterizations. Through experimental investigation, we find that DPPO takes advantage of unique synergies between RL fine-tuning and the diffusion parameterization, leading to structured and on-manifold exploration, stable training, and strong policy robustness. We further demonstrate the strengths of DPPO in a range of realistic settings, including simulated robotic tasks with pixel observations, and via zero-shot deployment of simulation-trained policies on robot hardware in a long-horizon, multi-stage manipulation task. Website with code: diffusion-ppo.github.io

Key facts. Cited by 255 papers in the map (data through 7 Oct 2026). Placed in Diffusion and flow policies for robotics, one of 14 directions in Diffusion models. Builds on 91 of its 111 references that are in the map. First posted on arXiv 1 Sep 2024.

Builds on

91 of 111 references.

Closest ancestors · 22

Same direction, newest first

  1. 2024Policy-Guided Diffusion9 Apr
  2. 2024Behavior Generation with Latent Actions5 Mar
  3. 2024Diffusion World Model: Future Modeling Beyond Step-by-Step Rollout for Offline Reinforcement Learning5 Feb
  4. 2023Learning a Diffusion Model Policy from Rewards via Q-Score Matching18 Dec
  5. 2023World Models via Policy-Guided Trajectory Diffusion13 Dec
  6. 2023Diffusion Model Alignment Using Direct Preference Optimization21 Nov
  7. 2023Diffusion Models for Reinforcement Learning: A Survey2 Nov
  8. 2023NoMaD: Goal Masked Diffusion Policies for Navigation and Exploration11 Oct
  9. 2023Boosting Continuous Control with Consistency Policy10 Oct
  10. 2023Directly Fine-Tuning Diffusion Models on Differentiable Rewards29 Sep
  11. 2023Consistency Models as a Rich and Efficient Policy Class for Reinforcement Learning29 Sep
  12. 2023Reasoning with Latent Diffusion in Offline Reinforcement Learning12 Sep
10 more
  1. 2023Efficient Diffusion Policies for Offline Reinforcement Learning31 May
  2. 2023Training Diffusion Models with Reinforcement Learning22 May
  3. 2023Policy Representation via Diffusion Probability Model for Reinforcement Learning22 May
  4. 2023IDQL: Implicit Q-Learning as an Actor-Critic Method with Diffusion Policies20 Apr
  5. 2023Goal-Conditioned Imitation Learning using Score-based Diffusion Policies5 Apr
  6. 2023Diffusion Policy: Visuomotor Policy Learning via Action Diffusion7 Mar
  7. 2023AdaptDiffuser: Diffusion Models as Adaptive Self-evolving Planners3 Feb
  8. 2023Imitating Human Behaviour with Diffusion Models25 Jan
  9. 2022Is Conditional Generative Modeling all you need for Decision-Making?28 Nov
  10. 2022Diffusion Policies as an Expressive Policy Class for Offline Reinforcement Learning12 Aug

Other references · 69

  1. 2017Attention Is All You Needcited by 59,932
  2. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  3. 2020Language Models are Few-Shot Learnerscited by 32,199
  4. 2020Denoising Diffusion Probabilistic Modelscited by 20,851
  5. 2015U-Net: Convolutional Networks for Biomedical Image Segmentationcited by 19,604
  6. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,694
  7. 2022Training language models to follow instructions with human feedbackcited by 15,712
  8. 2017Proximal Policy Optimization Algorithmscited by 14,441
  9. 2020Denoising Diffusion Implicit Modelscited by 9,655
  10. 2020Score-Based Generative Modeling through Stochastic Differential Equationscited by 8,964
  11. 2015Deep Unsupervised Learning using Nonequilibrium Thermodynamicscited by 7,005
  12. 2018Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actorcited by 4,635
38 more
  1. 2021Improved Denoising Diffusion Probabilistic Modelscited by 3,736
  2. 2022DreamBooth: Fine Tuning Text-to-Image Diffusion Models for Subject-Driven Generationcited by 3,286
  3. 2015Continuous control with deep reinforcement learningcited by 3,279
  4. 2022DreamFusion: Text-to-3D using 2D Diffusioncited by 3,006
  5. 2015High-Dimensional Continuous Control Using Generalized Advantage Estimationcited by 1,916
  6. 2022Imagen Video: High Definition Video Generation with Diffusion Modelscited by 1,765
  7. 2023Learning Fine-Grained Bimanual Manipulation with Low-Cost Hardwarecited by 1,490
  8. 2020D4RL: Datasets for Deep Data-Driven Reinforcement Learningcited by 1,406
  9. 2020DiffWave: A Versatile Diffusion Model for Audio Synthesiscited by 1,339
  10. 2022Planning with Diffusion for Flexible Behavior Synthesiscited by 1,023
  11. 2021Isaac Gym: High Performance GPU-Based Physics Simulation For Robot Learningcited by 882
  12. 2024Simple and Effective Masked Diffusion Language Modelscited by 800
  13. 2019Advantage-Weighted Regression: Simple and Scalable Off-Policy Reinforcement Learningcited by 750
  14. 2023Discrete Diffusion Modeling by Estimating the Ratios of the Data Distributioncited by 712
  15. 2024Genie: Generative Interactive Environmentscited by 706
  16. 2021What Matters in Learning from Offline Human Demonstrations for Robot Manipulationcited by 699
  17. 2017Learning Complex Dexterous Manipulation with Deep Reinforcement Learning and Demonstrationscited by 685
  18. 2020AWAC: Accelerating Online Reinforcement Learning with Offline Datasetscited by 647
  19. 2018Learning Dexterous In-Hand Manipulationcited by 643
  20. 2024Diffusion Forcing: Next-token Prediction Meets Full-Sequence Diffusioncited by 634
  21. 2018An Algorithmic Perspective on Imitation Learningcited by 455
  22. 2021Implicit Behavioral Cloningcited by 442
  23. 2023Learning Interactive Real-World Simulatorscited by 440
  24. 2019Learning agile and dynamic motor skills for legged robotscited by 402
  25. 2019Relay Policy Learning: Solving Long-Horizon Tasks via Imitation and Reinforcement Learningcited by 380
  26. 2024Mobile ALOHA: Learning Bimanual Mobile Manipulation with Low-Cost Whole-Body Teleoperationcited by 377
  27. 2023Efficient Online Reinforcement Learning with Offline Datacited by 332
  28. 2017Leveraging Demonstrations for Deep Reinforcement Learning on Robotics Problems with Sparse Rewardscited by 302
  29. 2023Cal-QL: Calibrated Offline RL Pre-Training for Efficient Online Fine-Tuningcited by 230
  30. 2021Offline-to-Online Reinforcement Learning via Balanced Replay and Pessimistic Q-Ensemblecited by 211
  31. 2022Offline Reinforcement Learning via High-Fidelity Generative Behavior Modelingcited by 177
  32. 2017Deep Reinforcement Learning for De-Novo Drug Designcited by 161
  33. 2021A System for General In-Hand Object Re-Orientationcited by 155
  34. 2024Reconciling Reality through Simulation: A Real-to-Sim-to-Real Approach for Robust Manipulationcited by 135
  35. 2023FurnitureBench: Reproducible Real-World Benchmark for Long-Horizon Complex Manipulationcited by 132
  36. 2024SERL: A Software Suite for Sample-Efficient Robotic Reinforcement Learningcited by 113
  37. 2019Self-Supervised Correspondence in Visuomotor Policy Learningcited by 106
  38. 2023Policy Expansion for Bridging Offline-to-Online Reinforcement Learningcited by 103

50 most cited of 69 All 69 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 13; 2025: 113; 2026: 129.

255 papers.

Sort:

By area:

  1. 2025π*0.6: a VLA That Learns From Experiencecited by 280
  2. 2025DexVLA: Vision-Language Model with Plug-In Diffusion Expert for General Robot Controlcited by 191
  3. 2025ReCogDrive: A Reinforced Cognitive Framework for End-to-End Autonomous Drivingcited by 190
  4. 2025DiffuCoder: Understanding and Improving Masked Diffusion Models for Code Generationcited by 184
  5. 2025World Simulation with Video Foundation Models for Physical AIcited by 183
  6. 2025Flow Q-Learningcited by 162
  7. 2025Steering Your Diffusion Policy with Latent Space Reinforcement Learningcited by 149
  8. 2025What Can RL Bring to VLA Generalization? An Empirical Studycited by 102
  9. 2025Reinforcement Learning with Action Chunkingcited by 97
  10. 2025Flow Matching Policy Gradientscited by 86
  11. 2024Policy Agnostic RL: Offline RL and Online RL Fine-Tuning of Any Class and Backbonecited by 71
  12. 2025A Vision-Language-Action-Critic Model for Robotic Real-World Reinforcement Learningcited by 66
38 more
  1. 2025Efficient Online Reinforcement Learning for Diffusion Policycited by 61
  2. 2024Policy Decorator: Model-Agnostic Online Refinement for Large Policy Modelcited by 60
  3. 2024Efficient Online Reinforcement Learning Fine-Tuning Need Not Retain Offline Datacited by 58
  4. 2025DiffusionDriveV2: Reinforcement Learning-Constrained Truncated Diffusion Modeling in End-to-End Autonomous Drivingcited by 52
  5. 2025Diffusion Guidance Is a Controllable Policy Improvement Operatorcited by 51
  6. 2025Inference-time Scaling of Diffusion Models through Classical Searchcited by 48
  7. 2025RL-100: Performant Robotic Manipulation with Real-World Reinforcement Learningcited by 47
  8. 2026RL Token: Bootstrapping Online RL with Vision-Language-Action Modelscited by 43
  9. 2025Residual Off-Policy RL for Finetuning Behavior Cloning Policiescited by 42
  10. 2025Diffusion Models for Robotic Manipulation: A Surveycited by 39
  11. 2025Fine-Tuning Discrete Diffusion Models with Policy Gradient Methodscited by 36
  12. 2025Much Ado About Noising: Dispelling the Myths of Generative Robotic Controlcited by 35
  13. 2025Score as Action: Fine-Tuning Diffusion Generative Models by Continuous-time Reinforcement Learningcited by 33
  14. 2025Flow-Based Policy for Online Reinforcement Learningcited by 33
  15. 2026Q-learning with Adjoint Matchingcited by 33
  16. 2025Diffusion Tree Sampling: Scalable inference-time alignment of diffusion modelscited by 31
  17. 2025IRL-VLA: Training an Vision-Language-Action Policy via Reward World Modelcited by 31
  18. 2025EXPO: Stable Reinforcement Learning with Expressive Policiescited by 30
  19. 2025From Forecasting to Planning: Policy World Model for Collaborative State-Action Predictioncited by 29
  20. 2025DiWA: Diffusion Policy Adaptation with World Modelscited by 28
  21. 2025Scaling Offline RL via Efficient and Expressive Shortcut Modelscited by 27
  22. 2025SAC Flow: Sample-Efficient Reinforcement Learning of Flow-Based Policies via Velocity-Reparameterized Sequential Modelingcited by 27
  23. 2024DOLLAR: Few-Step Video Generation via Distillation and Latent Reward Optimizationcited by 25
  24. 2025Spatial-Temporal Graph Diffusion Policy with Kinematic Modeling for Bimanual Robotic Manipulationcited by 24
  25. 2025AnyPos: Automated Task-Agnostic Actions for Bimanual Manipulationcited by 23
  26. 2025VITA: Vision-to-Action Flow Matching Policycited by 23
  27. 2024TidyBot++: An Open-Source Holonomic Mobile Manipulator for Robot Learningcited by 21
  28. 2024ManiSkill-HAB: A Benchmark for Low-Level Manipulation in Home Rearrangement Taskscited by 20
  29. 2025floq: Training Critics via Flow-Matching for Scaling Compute in Value-Based RLcited by 20
  30. 2025RFTF: Reinforcement Fine-tuning for Embodied Agents with Temporal Feedbackcited by 19
  31. 2025Behavior-Regularized Diffusion Policy Optimization for Offline Reinforcement Learningcited by 18
  32. 2025One-Step Flow Policy Mirror Descentcited by 18
  33. 2026Rethinking the Design Space of Reinforcement Learning for Diffusion Models: On the Importance of Likelihood Estimation Beyond Loss Designcited by 17
  34. 2025Learning Personalized Driving Styles via Reinforcement Learning from Human Feedbackcited by 16
  35. 2025GenPO: Generative Diffusion Models Meet On-Policy Reinforcement Learningcited by 16
  36. 2025A Smooth Sea Never Made a Skilled SAILOR: Robust Imitation via Learning to Searchcited by 16
  37. 2025π_textttRL: Online RL Fine-tuning for Flow-based Vision-Language-Action Modelscited by 16
  38. 2025FDPP: Fine-tune Diffusion Policy with Human Preferencecited by 14

50 most cited of 255 All 255 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2026From Noise to Control: Parameterized Diffusion Policies
  2. 2024Sparse Diffusion Policy: A Sparse, Reusable, and Flexible Policy for Robot Learning
  3. 2023Score Regularized Policy Optimization through Diffusion Behavior
  4. 2025Exploration and Adaptation in Non-Stationary Tasks with Diffusion Policies
  5. 2025Real-Time Iteration Scheme for Diffusion Policy
  6. 2024Enhancing Sample Efficiency and Exploration in Reinforcement Learning through the Integration of Diffusion Models and Proximal Policy Optimization
  7. 2025D2PPO: Diffusion Policy Policy Optimization with Dispersive Loss
  8. 2025D3P: Dynamic Denoising Diffusion Policy via Reinforcement Learning