# PBCS : Efficient Exploration and Exploitation Using a Synergy between Reinforcement Learning and Motion Planning

arXiv: https://arxiv.org/abs/2004.11667 · HTML: https://mlsanity.com/paper/2004.11667/

## Key facts

- Cited by 12 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 24 Apr 2020.

## Built on by (most cited first, 12 of 12)

- [Exploration in Deep Reinforcement Learning: A Survey](https://mlsanity.com/paper/2205.00824/) (2022)
- [Distilling Motion Planner Augmented Policies into Visual Control Policies for Robot Manipulation](https://mlsanity.com/paper/2111.06383/) (2021)
- [Generative Personas That Behave and Experience Like Humans](https://mlsanity.com/paper/2209.00459/) (2022)
- [Go-Blend behavior and affect](https://mlsanity.com/paper/2109.13388/) (2021)
- [Play with Emotion: Affect-Driven Reinforcement Learning](https://mlsanity.com/paper/2208.12622/) (2022)
- [Leveraging Sequentiality in Reinforcement Learning from a Single Demonstration](https://mlsanity.com/paper/2211.04786/) (2022)
- [Cell-Free Latent Go-Explore](https://mlsanity.com/paper/2208.14928/) (2022)
- [Imagine, Initialize, and Explore: An Effective Exploration Method in Multi-Agent Reinforcement Learning](https://mlsanity.com/paper/2402.17978/) (2024)
- [Divide & Conquer Imitation Learning](https://mlsanity.com/paper/2204.07404/) (2022)
- [Selection-Expansion: A Unifying Framework for Motion-Planning and Diversity Search Algorithms](https://mlsanity.com/paper/2104.04768/) (2021)
- [CalliRewrite: Recovering Handwriting Behaviors from Calligraphy Images without Supervision](https://mlsanity.com/paper/2405.15776/) (2024)
- [Bidirectional Voronoi-biased Exploration Curriculum for Reinforcement Learning](https://mlsanity.com/paper/2610.03395/) (2026)

## Abstract (arXiv)

The exploration-exploitation trade-off is at the heart of reinforcement learning (RL). However, most continuous control benchmarks used in recent RL research only require local exploration. This led to the development of algorithms that have basic exploration capabilities, and behave poorly in benchmarks that require more versatile exploration. For instance, as demonstrated in our empirical study, state-of-the-art RL algorithms such as DDPG and TD3 are unable to steer a point mass in even small 2D mazes. In this paper, we propose a new algorithm called "Plan, Backplay, Chain Skills" (PBCS) that combines motion planning and reinforcement learning to solve hard exploration environments. In a first phase, a motion planning algorithm is used to find a single good trajectory, then an RL algorithm is trained using a curriculum derived from the trajectory, by combining a variant of the Backplay algorithm and skill chaining. We show that this method outperforms state-of-the-art RL algorithms in 2D maze environments of various sizes, and is able to improve on the trajectory obtained by the motion planning phase.