# Self-Imitation Learning by Planning

arXiv: https://arxiv.org/abs/2103.13834 · HTML: https://mlsanity.com/paper/2103.13834/

## Key facts

- Cited by 26 papers in the map (data through 14 Oct 2026).
- Placed in Robot manipulation imitation learning, one of 13 directions in Offline RL and imitation learning from data.
- Builds on 20 of its 37 references that are in the map.
- First posted on arXiv 25 Mar 2021.

## Placed in

- Offline RL and imitation learning from data / [Robot manipulation imitation learning](https://mlsanity.com/offlinerl/robot-manipulation-imitation-learning/)

## Builds on (most cited first, 20 of 20)

- [Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actor](https://mlsanity.com/paper/1801.01290/) (2018)
- [Continuous control with deep reinforcement learning](https://mlsanity.com/paper/1509.02971/) (2015)
- [Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problems](https://mlsanity.com/paper/2005.01643/) (2020)
- [Soft Actor-Critic Algorithms and Applications](https://mlsanity.com/paper/1812.05905/) (2018)
- [Hindsight Experience Replay](https://mlsanity.com/paper/1707.01495/) (2017)
- [QT-Opt: Scalable Deep Reinforcement Learning for Vision-Based Robotic Manipulation](https://mlsanity.com/paper/1806.10293/) (2018)
- [Learning Complex Dexterous Manipulation with Deep Reinforcement Learning and Demonstrations](https://mlsanity.com/paper/1709.10087/) (2017)
- [Overcoming Exploration in Reinforcement Learning with Demonstrations](https://mlsanity.com/paper/1709.10089/) (2017)
- [Leveraging Demonstrations for Deep Reinforcement Learning on Robotics Problems with Sparse Rewards](https://mlsanity.com/paper/1707.08817/) (2017)
- [Imagination-Augmented Agents for Deep Reinforcement Learning](https://mlsanity.com/paper/1707.06203/) (2017)
- [Self-Imitation Learning](https://mlsanity.com/paper/1806.05635/) (2018)
- [Motion Planning Networks](https://mlsanity.com/paper/1806.05767/) (2018)
- [Learning Navigation Behaviors End-to-End with AutoRL](https://mlsanity.com/paper/1809.10124/) (2018)
- [ReLMoGen: Leveraging Motion Generation in Reinforcement Learning for Mobile Manipulation](https://mlsanity.com/paper/2008.07792/) (2020)
- [Harnessing Reinforcement Learning for Neural Motion Planning](https://mlsanity.com/paper/1906.00214/) (2019)
- [APPLD: Adaptive Planner Parameter Learning from Demonstration](https://mlsanity.com/paper/2004.00116/) (2020)
- [Meta-SAC: Auto-tune the Entropy Temperature of Soft Actor-Critic via Metagradient](https://mlsanity.com/paper/2007.01932/) (2020)
- [Accelerating Reinforcement Learning for Reaching using Continuous Curriculum Learning](https://mlsanity.com/paper/2002.02697/) (2020)
- [Using Local Experiences for Global Motion Planning](https://mlsanity.com/paper/1903.08693/) (2019)
- [Interactive Open-Ended Object, Affordance and Grasp Learning for Robotic Manipulation](https://mlsanity.com/paper/1904.02530/) (2019)

## Built on by (most cited first, 26 of 26)

- [End-to-end Autonomous Driving: Challenges and Frontiers](https://mlsanity.com/paper/2306.16927/) (2023)
- [Imitation Learning: Progress, Taxonomies and Challenges](https://mlsanity.com/paper/2106.12177/) (2021)
- [Task2Sim : Towards Effective Pre-training and Transfer from Synthetic Data](https://mlsanity.com/paper/2112.00054/) (2021)
- [Off-Policy Reinforcement Learning with Delayed Rewards](https://mlsanity.com/paper/2106.11854/) (2021)
- [Phasic Self-Imitative Reduction for Sparse-Reward Goal-Conditioned Reinforcement Learning](https://mlsanity.com/paper/2206.12030/) (2022)
- [Imitating Graph-Based Planning with Goal-Conditioned Policies](https://mlsanity.com/paper/2303.11166/) (2023)
- [Learn the Ropes, Then Trust the Wins: Self-imitation with Progressive Exploration for Agentic Reinforcement Learning](https://mlsanity.com/paper/2509.22601/) (2025)
- [Distilling Motion Planner Augmented Policies into Visual Control Policies for Robot Manipulation](https://mlsanity.com/paper/2111.06383/) (2021)
- [Optimistic Reinforcement Learning by Forward Kullback-Leibler Divergence Optimization](https://mlsanity.com/paper/2105.12991/) (2021)
- [Evolutionary Action Selection for Gradient-based Policy Learning](https://mlsanity.com/paper/2201.04286/) (2022)
- [Visual Hindsight Self-Imitation Learning for Interactive Navigation](https://mlsanity.com/paper/2312.03446/) (2023)
- [Learning from demonstrations with SACR2: Soft Actor-Critic with Reward Relabeling](https://mlsanity.com/paper/2110.14464/) (2021)
- [Reinforcement learning with Demonstrations from Mismatched Task under Sparse Reward](https://mlsanity.com/paper/2212.01509/) (2022)
- [Reinforcement Learning in Robotic Motion Planning by Combined Experience-based Planning and Self-Imitation Learning](https://mlsanity.com/paper/2306.06754/) (2023)
- [Emerging Threats in Deep Learning-Based Autonomous Driving: A Comprehensive Survey](https://mlsanity.com/paper/2210.11237/) (2022)
- [Towards Improving Exploration in Self-Imitation Learning using Intrinsic Motivation](https://mlsanity.com/paper/2211.16838/) (2022)
- [Co-Imitation Learning without Expert Demonstration](https://mlsanity.com/paper/2103.14823/) (2021)
- [Shortest-Path Constrained Reinforcement Learning for Sparse Reward Tasks](https://mlsanity.com/paper/2107.06405/) (2021)
- [Robust Action Gap Increasing with Clipped Advantage Learning](https://mlsanity.com/paper/2203.11677/) (2022)
- [Dual Behavior Regularized Reinforcement Learning](https://mlsanity.com/paper/2109.09037/) (2021)
- [A Fully Controllable Agent in the Path Planning using Goal-Conditioned Reinforcement Learning](https://mlsanity.com/paper/2205.09967/) (2022)
- [A Parametric Class of Approximate Gradient Updates for Policy Optimization](https://mlsanity.com/paper/2206.08499/) (2022)
- [SIL-RRT*: Learning Sampling Distribution through Self Imitation Learning](https://mlsanity.com/paper/2411.17293/) (2024)
- [Prompt Informed Reinforcement Learning for Visual Coverage Path Planning](https://mlsanity.com/paper/2507.10284/) (2025)
- [Match or Replay: Self Imitating Proximal Policy Optimization](https://mlsanity.com/paper/2603.27515/) (2026)
- [Temporal Self-Imitation Learning](https://mlsanity.com/paper/2606.19752/) (2026)

## Abstract (arXiv)

Imitation learning (IL) enables robots to acquire skills quickly by transferring expert knowledge, which is widely adopted in reinforcement learning (RL) to initialize exploration. However, in long-horizon motion planning tasks, a challenging problem in deploying IL and RL methods is how to generate and collect massive, broadly distributed data such that these methods can generalize effectively. In this work, we solve this problem using our proposed approach called {self-imitation learning by planning (SILP)}, where demonstration data are collected automatically by planning on the visited states from the current policy. SILP is inspired by the observation that successfully visited states in the early reinforcement learning stage are collision-free nodes in the graph-search based motion planner, so we can plan and relabel robot's own trials as demonstrations for policy learning. Due to these self-generated demonstrations, we relieve the human operator from the laborious data preparation process required by IL and RL methods in solving complex motion planning tasks. The evaluation results show that our SILP method achieves higher success rates and enhances sample efficiency compared to selected baselines, and the policy learned in simulation performs well in a real-world placement task with changing goals and obstacles.