# Hierarchical Policies for Cluttered-Scene Grasping with Latent Plans

arXiv: https://arxiv.org/abs/2107.01518 · HTML: https://mlsanity.com/paper/2107.01518/

## Key facts

- Cited by 12 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 4 Jul 2021.

## Built on by (most cited first, 12 of 12)

- [What Matters in Language Conditioned Robotic Imitation Learning over Unstructured Data](https://mlsanity.com/paper/2204.06252/) (2022)
- [Policy Representation via Diffusion Probability Model for Reinforcement Learning](https://mlsanity.com/paper/2305.13122/) (2023)
- ["This is my unicorn, Fluffy": Personalizing frozen vision-language representations](https://mlsanity.com/paper/2204.01694/) (2022)
- [Latent Plans for Task-Agnostic Offline Reinforcement Learning](https://mlsanity.com/paper/2209.08959/) (2022)
- [NeRF in the Palm of Your Hand: Corrective Augmentation for Robotics via Novel-View Synthesis](https://mlsanity.com/paper/2301.08556/) (2023)
- [Inference-Time Policy Steering through Human Interactions](https://mlsanity.com/paper/2411.16627/) (2024)
- [Where To Start? Transferring Simple Skills to Complex Environments](https://mlsanity.com/paper/2212.06111/) (2022)
- [Cluttered Food Grasping with Adaptive Fingers and Synthetic-Data Trained Object Detection](https://mlsanity.com/paper/2203.05187/) (2022)
- [Task and Motion Planning for Execution in the Real](https://mlsanity.com/paper/2406.03641/) (2024)
- [Learning Any-View 6DoF Robotic Grasping in Cluttered Scenes via Neural Surface Rendering](https://mlsanity.com/paper/2306.07392/) (2023)
- [Learning Pneumatic Non-Prehensile Manipulation with a Mobile Blower](https://mlsanity.com/paper/2204.02390/) (2022)
- [SE(3) Neural Potential Fields for 6-DoF Trajectory Planning Directly from Images Without Explicit 3D Reconstruction](https://mlsanity.com/paper/2609.24864/) (2026)

## Abstract (arXiv)

6D grasping in cluttered scenes is a longstanding problem in robotic manipulation. Open-loop manipulation pipelines may fail due to inaccurate state estimation, while most end-to-end grasping methods have not yet scaled to complex scenes with obstacles. In this work, we propose a new method for end-to-end learning of 6D grasping in cluttered scenes. Our hierarchical framework learns collision-free target-driven grasping based on partial point cloud observations. We learn an embedding space to encode expert grasping plans during training and a variational autoencoder to sample diverse grasping trajectories at test time. Furthermore, we train a critic network for plan selection and an option classifier for switching to an instance grasping policy through hierarchical reinforcement learning. We evaluate our method and compare against several baselines in simulation, as well as demonstrate that our latent planning can generalize to real-world cluttered-scene grasping tasks. Our videos and code can be found at https://sites.google.com/view/latent-grasping .