# Reinforcement Learning in Robotic Motion Planning by Combined Experience-based Planning and Self-Imitation Learning

arXiv: https://arxiv.org/abs/2306.06754 · HTML: https://mlsanity.com/paper/2306.06754/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- Placed in Robot manipulation imitation learning, one of 13 directions in Offline RL and imitation learning from data.
- Builds on 32 of its 73 references that are in the map.
- First posted on arXiv 11 Jun 2023.

## Placed in

- Offline RL and imitation learning from data / [Robot manipulation imitation learning](https://mlsanity.com/offlinerl/robot-manipulation-imitation-learning/)

## Builds on (most cited first, 30 of 32)

- [Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actor](https://mlsanity.com/paper/1801.01290/) (2018)
- [Continuous control with deep reinforcement learning](https://mlsanity.com/paper/1509.02971/) (2015)
- [Off-Policy Deep Reinforcement Learning without Exploration](https://mlsanity.com/paper/1812.02900/) (2018)
- [Soft Actor-Critic Algorithms and Applications](https://mlsanity.com/paper/1812.05905/) (2018)
- [Hindsight Experience Replay](https://mlsanity.com/paper/1707.01495/) (2017)
- [QT-Opt: Scalable Deep Reinforcement Learning for Vision-Based Robotic Manipulation](https://mlsanity.com/paper/1806.10293/) (2018)
- [Learning Complex Dexterous Manipulation with Deep Reinforcement Learning and Demonstrations](https://mlsanity.com/paper/1709.10087/) (2017)
- [Overcoming Exploration in Reinforcement Learning with Demonstrations](https://mlsanity.com/paper/1709.10089/) (2017)
- [Leveraging Demonstrations for Deep Reinforcement Learning on Robotics Problems with Sparse Rewards](https://mlsanity.com/paper/1707.08817/) (2017)
- [Search on the Replay Buffer: Bridging Planning and Reinforcement Learning](https://mlsanity.com/paper/1906.05253/) (2019)
- [Learning Continuous Control Policies by Stochastic Value Gradients](https://mlsanity.com/paper/1510.09142/) (2015)
- [Imagination-Augmented Agents for Deep Reinforcement Learning](https://mlsanity.com/paper/1707.06203/) (2017)
- [Self-Imitation Learning](https://mlsanity.com/paper/1806.05635/) (2018)
- [Learning Sampling Distributions for Robot Motion Planning](https://mlsanity.com/paper/1709.05448/) (2017)
- [Motion Planning Networks](https://mlsanity.com/paper/1806.05767/) (2018)
- [Safe Reinforcement Learning with Model Uncertainty Estimates](https://mlsanity.com/paper/1810.08700/) (2018)
- [Learning Navigation Behaviors End-to-End with AutoRL](https://mlsanity.com/paper/1809.10124/) (2018)
- [Long-Range Indoor Navigation with PRM-RL](https://mlsanity.com/paper/1902.09458/) (2019)
- [Where to go next: Learning a Subgoal Recommendation Policy for Navigation Among Pedestrians](https://mlsanity.com/paper/2102.13073/) (2021)
- [Model Predictive Actor-Critic: Accelerating Robot Skill Acquisition with Deep Reinforcement Learning](https://mlsanity.com/paper/2103.13842/) (2021)
- [Self-Imitation Learning by Planning](https://mlsanity.com/paper/2103.13834/) (2021)
- [Reinforcement Learning from Imperfect Demonstrations under Soft Expert Guidance](https://mlsanity.com/paper/1911.07109/) (2019)
- [Harnessing Reinforcement Learning for Neural Motion Planning](https://mlsanity.com/paper/1906.00214/) (2019)
- [APPLD: Adaptive Planner Parameter Learning from Demonstration](https://mlsanity.com/paper/2004.00116/) (2020)
- [Learning Implicit Sampling Distributions for Motion Planning](https://mlsanity.com/paper/1806.01968/) (2018)
- [Experience-Based Planning with Sparse Roadmap Spanners](https://mlsanity.com/paper/1410.1950/) (2014)
- [Episodic Self-Imitation Learning with Hindsight](https://mlsanity.com/paper/2011.13467/) (2020)
- [Using Local Experiences for Global Motion Planning](https://mlsanity.com/paper/1903.08693/) (2019)
- [Policy Optimization to Learn Adaptive Motion Primitives in Path Planning with Dynamic Obstacles](https://mlsanity.com/paper/2212.14307/) (2022)
- [Singularity-free Guiding Vector Field for Robot Navigation](https://mlsanity.com/paper/2012.01826/) (2020)

## Built on by (most cited first, 4 of 4)

- [Toward Generalist Neural Motion Planners for Robotic Manipulators: Challenges and Opportunities](https://mlsanity.com/paper/2603.24318/) (2026)
- [Visualizing High-Dimensional Configuration Spaces: A Comprehensive Analytical Approach](https://mlsanity.com/paper/2312.10918/) (2023)
- [Match or Replay: Self Imitating Proximal Policy Optimization](https://mlsanity.com/paper/2603.27515/) (2026)
- [Temporal Self-Imitation Learning](https://mlsanity.com/paper/2606.19752/) (2026)

## Abstract (arXiv)

High-quality and representative data is essential for both Imitation Learning (IL)- and Reinforcement Learning (RL)-based motion planning tasks. For real robots, it is challenging to collect enough qualified data either as demonstrations for IL or experiences for RL due to safety considerations in environments with obstacles. We target this challenge by proposing the self-imitation learning by planning plus (SILP+) algorithm, which efficiently embeds experience-based planning into the learning architecture to mitigate the data-collection problem. The planner generates demonstrations based on successfully visited states from the current RL policy, and the policy improves by learning from these demonstrations. In this way, we relieve the demand for human expert operators to collect demonstrations required by IL and improve the RL performance as well. Various experimental results show that SILP+ achieves better training efficiency higher and more stable success rate in complex motion planning tasks compared to several other methods. Extensive tests on physical robots illustrate the effectiveness of SILP+ in a physical setting.