# Push Smarter, Not Harder: Hierarchical RL-Diffusion Policy for Efficient Nonprehensile Manipulation

arXiv: https://arxiv.org/abs/2512.10099 · HTML: https://mlsanity.com/paper/2512.10099/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Placed in Diffusion and flow policies for robotics, one of 14 directions in Diffusion models.
- Builds on 24 of its 33 references that are in the map.
- First posted on arXiv 10 Dec 2025.

## Placed in

- Diffusion models / [Diffusion and flow policies for robotics](https://mlsanity.com/diffusion/diffusion-and-flow-policies-for-robotics/)
- Deep reinforcement learning methods / [RL for robot locomotion and manipulation](https://mlsanity.com/deeprl/rl-for-robot-locomotion-and-manipulation/)

## Builds on (most cited first, 24 of 24)

- [Denoising Diffusion Probabilistic Models](https://mlsanity.com/paper/2006.11239/) (2020)
- [Deep Unsupervised Learning using Nonequilibrium Thermodynamics](https://mlsanity.com/paper/1503.03585/) (2015)
- [Playing Atari with Deep Reinforcement Learning](https://mlsanity.com/paper/1312.5602/) (2013)
- [Diffusion Policy: Visuomotor Policy Learning via Action Diffusion](https://mlsanity.com/paper/2303.04137/) (2023)
- [FiLM: Visual Reasoning with a General Conditioning Layer](https://mlsanity.com/paper/1709.07871/) (2017)
- [Deep Reinforcement Learning with Double Q-learning](https://mlsanity.com/paper/1509.06461/) (2015)
- [Planning with Diffusion for Flexible Behavior Synthesis](https://mlsanity.com/paper/2205.09991/) (2022)
- [FeUdal Networks for Hierarchical Reinforcement Learning](https://mlsanity.com/paper/1703.01161/) (2017)
- [Accelerating Reinforcement Learning with Learned Skill Priors](https://mlsanity.com/paper/2010.11944/) (2020)
- [Hierarchical Diffusion Policy for Kinematics-Aware Multi-Task Robotic Manipulation](https://mlsanity.com/paper/2403.03890/) (2024)
- [Simple Hierarchical Planning with Diffusion](https://mlsanity.com/paper/2401.02644/) (2024)
- [Spatial Action Maps for Mobile Manipulation](https://mlsanity.com/paper/2004.09141/) (2020)
- [On the Role of the Action Space in Robot Manipulation Learning and Sim-to-Real Transfer](https://mlsanity.com/paper/2312.03673/) (2023)
- [Let's Push Things Forward: A Survey on Robot Pushing](https://mlsanity.com/paper/1905.05138/) (2019)
- [Spatial Intention Maps for Multi-Agent Mobile Manipulation](https://mlsanity.com/paper/2103.12710/) (2021)
- [Rearrangement with Nonprehensile Manipulation Using Deep Reinforcement Learning](https://mlsanity.com/paper/1803.05752/) (2018)
- [Nonprehensile Planar Manipulation through Reinforcement Learning with Multimodal Categorical Exploration](https://mlsanity.com/paper/2308.02459/) (2023)
- [A Data-Efficient Approach to Precise and Controlled Pushing](https://mlsanity.com/paper/1807.09904/) (2018)
- [Diffusion-Reinforcement Learning Hierarchical Motion Planning in Multi-agent Adversarial Games](https://mlsanity.com/paper/2403.10794/) (2024)
- [Hierarchical Reinforcement Learning with Uncertainty-Guided Diffusional Subgoals](https://mlsanity.com/paper/2505.21750/) (2025)
- [Pushing in the Dark: A Reactive Pushing Strategy for Mobile Robots Using Tactile Feedback](https://mlsanity.com/paper/2403.09305/) (2024)
- [Unwieldy Object Delivery with Nonholonomic Mobile Base: A Stable Pushing Approach](https://mlsanity.com/paper/2309.14295/) (2023)
- [Learning Pneumatic Non-Prehensile Manipulation with a Mobile Blower](https://mlsanity.com/paper/2204.02390/) (2022)
- [Bench-NPIN: Benchmarking Non-prehensile Interactive Navigation](https://mlsanity.com/paper/2505.12084/) (2025)

## Abstract (arXiv)

Nonprehensile manipulation, such as pushing objects across cluttered environments, presents a challenging control problem due to complex contact dynamics and long-horizon planning requirements. In this work, we propose HeRD, a hierarchical reinforcement learning-diffusion policy that decomposes pushing tasks into two levels: high-level goal selection and low-level trajectory generation. We employ a high-level reinforcement learning (RL) agent to select intermediate spatial goals, and a low-level goal-conditioned diffusion model to generate feasible, efficient trajectories to reach them. This architecture combines the long-term reward maximizing behaviour of RL with the generative capabilities of diffusion models. We evaluate our method in a 2D simulation environment and show that it outperforms the state-of-the-art baseline in success rate, path efficiency, and generalization across multiple environment configurations. Our results suggest that hierarchical control with generative low-level planning is a promising direction for scalable, goal-directed nonprehensile manipulation. Code, documentation, and trained models are available: https://github.com/carosteven/HeRD.