mlsanityData to 7 Oct 2026

A Single Diffusion-Policy Controller for Multi-Task Block Pushing with Zero-Shot Sim-to-Real Transfer

arXiv 2607.10892 ↗First posted 12 Jul 2026cs.ROcited by 0

On the map: Diffusion models → Diffusion and flow policies for robotics

Abstract (arXiv)

Diffusion policies have shown promising empirical performance in representing and learning complex maneuvers for robots using behavior cloning (BC). In this paper, we explore training diffusion policies from scratch using reinforcement learning (RL) for multi-task robotic manipulation. Specifically, we aim to train a single diffusion policy for block-pushing tasks with multiple shapes. The proposed framework features a simple policy loss function, which is a reweighted evidence lower bound used in BC-based diffusion policy training and can seamlessly serve as the policy learning module in RL algorithms. To address the exploration challenges arising from the absence of demonstrations, we incorporate reverse curriculum generation and objective-centric representations. Combined with the expressiveness of diffusion policies, our design supports learning of multi-task block-pushing policies in our sparse-reward simulation setting. We further evaluate whether the trained diffusion policy transfers in zero-shot to real-world tasks under varying environmental conditions including goal positions, block shapes, block weights and surface friction, providing evidence that this pipeline can transfer to our real-world block-pushing setup under the tested variations.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Placed in Diffusion and flow policies for robotics, one of 14 directions in Diffusion models. Builds on 41 of its 45 references that are in the map. First posted on arXiv 12 Jul 2026.

Builds on

41 of 45 references.

Closest ancestors · 15

Same direction, newest first

  1. 2025Energy-Weighted Flow Matching for Offline Reinforcement Learning6 Mar
  2. 2024Diffusion Policy Policy Optimization1 Sep
  3. 2024Diffusion Spectral Representation for Reinforcement Learning23 Jun
  4. 2024Streaming Diffusion Policy: Fast Policy Synthesis with Variable Noise Diffusion Models7 Jun
  5. 2024Diffusion-based Reinforcement Learning via Q-weighted Variational Policy Optimization25 May
  6. 2024Diffusion Actor-Critic with Entropy Regulator24 May
  7. 2024Hierarchical Diffusion Policy for Kinematics-Aware Multi-Task Robotic Manipulation6 Mar
  8. 2023Learning a Diffusion Model Policy from Rewards via Q-Score Matching18 Dec
  9. 2023Crossway Diffusion: Improving Diffusion-based Visuomotor Policy via Self-supervised Learning4 Jul
  10. 2023MetaDiffuser: Diffusion Model as Conditional Planner for Offline Meta-RL31 May
  11. 2023Diffusion Model is an Effective Planner and Data Synthesizer for Multi-Task Reinforcement Learning29 May
  12. 2023Policy Representation via Diffusion Probability Model for Reinforcement Learning22 May
3 more
  1. 2023IDQL: Implicit Q-Learning as an Actor-Critic Method with Diffusion Policies20 Apr
  2. 2023Diffusion Policy: Visuomotor Policy Learning via Action Diffusion7 Mar
  3. 2022Diffusion Policies as an Expressive Policy Class for Offline Reinforcement Learning12 Aug

Other references · 26

  1. 2020Denoising Diffusion Probabilistic Modelscited by 20,851
  2. 2017Proximal Policy Optimization Algorithmscited by 14,441
  3. 2020Score-Based Generative Modeling through Stochastic Differential Equationscited by 8,964
  4. 2021Diffusion Models Beat GANs on Image Synthesiscited by 8,185
  5. 2015Deep Unsupervised Learning using Nonequilibrium Thermodynamicscited by 7,005
  6. 2018Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actorcited by 4,635
  7. 2015Trust Region Policy Optimizationcited by 2,503
  8. 2018Addressing Function Approximation Error in Actor-Critic Methodscited by 2,302
  9. 2017Domain Randomization for Transferring Deep Neural Networks from Simulation to the Real Worldcited by 1,401
  10. 2022Planning with Diffusion for Flexible Behavior Synthesiscited by 1,023
  11. 2018IMPALA: Scalable Distributed Deep-RL with Importance Weighted Actor-Learner Architecturescited by 866
  12. 2019Advantage-Weighted Regression: Simple and Scalable Off-Policy Reinforcement Learningcited by 750
14 more
  1. 2020AWAC: Accelerating Online Reinforcement Learning with Offline Datasetscited by 647
  2. 2023Learning Universal Policies via Text-Guided Video Generationcited by 506
  3. 2021Implicit Behavioral Cloningcited by 442
  4. 2020Curriculum Learning for Reinforcement Learning Domains: A Framework and Surveycited by 350
  5. 2022Contrastive Learning as Goal-Conditioned Reinforcement Learningcited by 241
  6. 2017Reverse Curriculum Generation for Reinforcement Learningcited by 185
  7. 2022Offline Reinforcement Learning via High-Fidelity Generative Behavior Modelingcited by 177
  8. 2022Goal-Conditioned Reinforcement Learning: Problems and Solutionscited by 145
  9. 2021Goal-Conditioned Reinforcement Learning with Imagined Subgoalscited by 129
  10. 2022Walk These Ways: Tuning Robot Control for Generalization with Multiplicity of Behaviorcited by 113
  11. 2025ASAP: Aligning Simulation and Real-World Physics for Learning Agile Humanoid Whole-Body Skillscited by 104
  12. 2020Mirror Descent Policy Optimizationcited by 102
  13. 2018Learning Curriculum Policies for Reinforcement Learningcited by 34
  14. 2023Learning to Fly in Secondscited by 18

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2025Steering Your Diffusion Policy with Latent Space Reinforcement Learning
  2. 2026A Review of Online Diffusion Policy RL Algorithms for Scalable Robotic Control
  3. 2025DiffusionRL: Efficient Training of Diffusion Policies for Robotic Grasping Using RL-Adapted Large-Scale Datasets
  4. 2025World4RL: Diffusion World Models for Policy Refinement with Reinforcement Learning for Robotic Manipulation
  5. 2024Unpacking the Individual Components of Diffusion Policy
  6. 2026Abstracting Robot Manipulation Skills via Mixture-of-Experts Diffusion Policies
  7. 2024One-Step Diffusion Policy: Fast Visuomotor Policies via Diffusion Distillation
  8. 2026Learning Diffusion Policy from Primitive Skills for Robot Manipulation