# Distilling Motion Planner Augmented Policies into Visual Control Policies for Robot Manipulation

arXiv: https://arxiv.org/abs/2111.06383 · HTML: https://mlsanity.com/paper/2111.06383/

## Key facts

- Cited by 10 papers in the map (data through 14 Oct 2026).
- Placed in Robot manipulation imitation learning, one of 13 directions in Offline RL and imitation learning from data.
- Builds on 27 of its 37 references that are in the map.
- First posted on arXiv 11 Nov 2021.

## Placed in

- Offline RL and imitation learning from data / [Robot manipulation imitation learning](https://mlsanity.com/offlinerl/robot-manipulation-imitation-learning/)

## Builds on (most cited first, 27 of 27)

- [Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actor](https://mlsanity.com/paper/1801.01290/) (2018)
- [Domain Randomization for Transferring Deep Neural Networks from Simulation to the Real World](https://mlsanity.com/paper/1703.06907/) (2017)
- [Hindsight Experience Replay](https://mlsanity.com/paper/1707.01495/) (2017)
- [Learning Complex Dexterous Manipulation with Deep Reinforcement Learning and Demonstrations](https://mlsanity.com/paper/1709.10087/) (2017)
- [AWAC: Accelerating Online Reinforcement Learning with Offline Datasets](https://mlsanity.com/paper/2006.09359/) (2020)
- [Overcoming Exploration in Reinforcement Learning with Demonstrations](https://mlsanity.com/paper/1709.10089/) (2017)
- [Leveraging Demonstrations for Deep Reinforcement Learning on Robotics Problems with Sparse Rewards](https://mlsanity.com/paper/1707.08817/) (2017)
- [Asymmetric Actor Critic for Image-Based Robot Learning](https://mlsanity.com/paper/1710.06542/) (2017)
- [Motion Planning Networks](https://mlsanity.com/paper/1806.05767/) (2018)
- [Beyond Pick-and-Place: Tackling Robotic Stacking of Diverse Shapes](https://mlsanity.com/paper/2110.06192/) (2021)
- [IKEA Furniture Assembly Environment for Long-Horizon Complex Manipulation Tasks](https://mlsanity.com/paper/1911.07246/) (2019)
- [ReLMoGen: Leveraging Motion Generation in Reinforcement Learning for Mobile Manipulation](https://mlsanity.com/paper/2008.07792/) (2020)
- [From Perception to Decision: A Data-driven Approach to End-to-end Motion Planning for Autonomous Ground Robots](https://mlsanity.com/paper/1609.07910/) (2016)
- [Self-Imitation Learning by Planning](https://mlsanity.com/paper/2103.13834/) (2021)
- [Motion Planner Augmented Reinforcement Learning for Robot Manipulation in Obstructed Environments](https://mlsanity.com/paper/2010.11940/) (2020)
- [Harnessing Reinforcement Learning for Neural Motion Planning](https://mlsanity.com/paper/1906.00214/) (2019)
- [Learning a Decentralized Multi-arm Motion Planner](https://mlsanity.com/paper/2011.02608/) (2020)
- [Deeply Informed Neural Sampling for Robot Motion Planning](https://mlsanity.com/paper/1809.10252/) (2018)
- [Self-Supervised Sim-to-Real Adaptation for Visual Robotic Manipulation](https://mlsanity.com/paper/1910.09470/) (2019)
- [Integrating Behavior Cloning and Reinforcement Learning for Improved Performance in Dense and Sparse Reward Environments](https://mlsanity.com/paper/1910.04281/) (2019)
- [Pre-training Neural Networks with Human Demonstrations for Deep Reinforcement Learning](https://mlsanity.com/paper/1709.04083/) (2017)
- [Meta-SAC: Auto-tune the Entropy Temperature of Soft Actor-Critic via Metagradient](https://mlsanity.com/paper/2007.01932/) (2020)
- [Learning Obstacle Representations for Neural Motion Planning](https://mlsanity.com/paper/2008.11174/) (2020)
- [PBCS : Efficient Exploration and Exploitation Using a Synergy between Reinforcement Learning and Motion Planning](https://mlsanity.com/paper/2004.11667/) (2020)
- [Policy Transfer across Visual and Dynamics Domain Gaps via Iterative Grounding](https://mlsanity.com/paper/2107.00339/) (2021)
- [Efficient Self-Supervised Data Collection for Offline Robot Learning](https://mlsanity.com/paper/2105.04607/) (2021)
- [A Convex Optimization Approach to Smooth Trajectories for Motion Planning with Car-Like Robots](https://mlsanity.com/paper/1506.01085/) (2015)

## Built on by (most cited first, 10 of 10)

- [Plan-Seq-Learn: Language Model Guided RL for Solving Long Horizon Robotics Tasks](https://mlsanity.com/paper/2405.01534/) (2024)
- [Learning Deformable Object Manipulation from Expert Demonstrations](https://mlsanity.com/paper/2207.10148/) (2022)
- [TWIST: Teacher-Student World Model Distillation for Efficient Sim-to-Real Transfer](https://mlsanity.com/paper/2311.03622/) (2023)
- [Proximal Policy Distillation](https://mlsanity.com/paper/2407.15134/) (2024)
- [Learning Robot Manipulation from Cross-Morphology Demonstration](https://mlsanity.com/paper/2304.03833/) (2023)
- [TraKDis: A Transformer-based Knowledge Distillation Approach for Visual Reinforcement Learning with Application to Cloth Manipulation](https://mlsanity.com/paper/2401.13362/) (2024)
- [Jacta: A Versatile Planner for Learning Dexterous and Whole-body Manipulation](https://mlsanity.com/paper/2408.01258/) (2024)
- [PLANRL: A Motion Planning and Imitation Learning Framework to Bootstrap Reinforcement Learning](https://mlsanity.com/paper/2408.04054/) (2024)
- [Decoupling Skill Learning from Robotic Control for Generalizable Object Manipulation](https://mlsanity.com/paper/2303.04016/) (2023)
- [RoboArm-NMP: a Learning Environment for Neural Motion Planning](https://mlsanity.com/paper/2405.16335/) (2024)

## Abstract (arXiv)

Learning complex manipulation tasks in realistic, obstructed environments is a challenging problem due to hard exploration in the presence of obstacles and high-dimensional visual observations. Prior work tackles the exploration problem by integrating motion planning and reinforcement learning. However, the motion planner augmented policy requires access to state information, which is often not available in the real-world settings. To this end, we propose to distill a state-based motion planner augmented policy to a visual control policy via (1) visual behavioral cloning to remove the motion planner dependency along with its jittery motion, and (2) vision-based reinforcement learning with the guidance of the smoothed trajectories from the behavioral cloning agent. We evaluate our method on three manipulation tasks in obstructed environments and compare it against various reinforcement learning and imitation learning baselines. The results demonstrate that our framework is highly sample-efficient and outperforms the state-of-the-art algorithms. Moreover, coupled with domain randomization, our policy is capable of zero-shot transfer to unseen environment settings with distractors. Code and videos are available at https://clvrai.com/mopa-pd