mlsanityData to 7 Oct 2026

Self-Imitation Learning by Planning

arXiv 2103.13834 ↗First posted 25 Mar 2021cs.RO cs.AIcited by 25

On the map: Offline RL and imitation learning from data → Robot manipulation imitation learning

Abstract (arXiv)

Imitation learning (IL) enables robots to acquire skills quickly by transferring expert knowledge, which is widely adopted in reinforcement learning (RL) to initialize exploration. However, in long-horizon motion planning tasks, a challenging problem in deploying IL and RL methods is how to generate and collect massive, broadly distributed data such that these methods can generalize effectively. In this work, we solve this problem using our proposed approach called {self-imitation learning by planning (SILP)}, where demonstration data are collected automatically by planning on the visited states from the current policy. SILP is inspired by the observation that successfully visited states in the early reinforcement learning stage are collision-free nodes in the graph-search based motion planner, so we can plan and relabel robot's own trials as demonstrations for policy learning. Due to these self-generated demonstrations, we relieve the human operator from the laborious data preparation process required by IL and RL methods in solving complex motion planning tasks. The evaluation results show that our SILP method achieves higher success rates and enhances sample efficiency compared to selected baselines, and the policy learned in simulation performs well in a real-world placement task with changing goals and obstacles.

Key facts. Cited by 25 papers in the map (data through 7 Oct 2026). Placed in Robot manipulation imitation learning, one of 13 directions in Offline RL and imitation learning from data. Builds on 20 of its 37 references that are in the map. First posted on arXiv 25 Mar 2021.

Builds on

20 of 37 references.

References · 20

  1. 2018Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actorcited by 4,635
  2. 2015Continuous control with deep reinforcement learningcited by 3,279
  3. 2020Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problemscited by 1,873
  4. 2018Soft Actor-Critic Algorithms and Applicationscited by 1,179
  5. 2017Hindsight Experience Replaycited by 1,096
  6. 2018QT-Opt: Scalable Deep Reinforcement Learning for Vision-Based Robotic Manipulationcited by 730
  7. 2017Learning Complex Dexterous Manipulation with Deep Reinforcement Learning and Demonstrationscited by 685
  8. 2017Overcoming Exploration in Reinforcement Learning with Demonstrationscited by 322
  9. 2017Leveraging Demonstrations for Deep Reinforcement Learning on Robotics Problems with Sparse Rewardscited by 302
  10. 2017Imagination-Augmented Agents for Deep Reinforcement Learningcited by 137
  11. 2018Self-Imitation Learningcited by 135
  12. 2018Motion Planning Networkscited by 78
8 more
  1. 2018Learning Navigation Behaviors End-to-End with AutoRLcited by 70
  2. 2020ReLMoGen: Leveraging Motion Generation in Reinforcement Learning for Mobile Manipulationcited by 47
  3. 2019Harnessing Reinforcement Learning for Neural Motion Planningcited by 19
  4. 2020APPLD: Adaptive Planner Parameter Learning from Demonstrationcited by 17
  5. 2020Meta-SAC: Auto-tune the Entropy Temperature of Soft Actor-Critic via Metagradientcited by 14
  6. 2020Accelerating Reinforcement Learning for Reaching using Continuous Curriculum Learningcited by 13
  7. 2019Using Local Experiences for Global Motion Planningcited by 8
  8. 2019Interactive Open-Ended Object, Affordance and Grasp Learning for Robotic Manipulationcited by 6

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 9; 2022: 8; 2023: 4; 2024: 1; 2025: 1; 2026: 2.

25 papers.

Sort:

By area:

  1. 2023End-to-end Autonomous Driving: Challenges and Frontierscited by 485
  2. 2021Imitation Learning: Progress, Taxonomies and Challengescited by 56
  3. 2021Task2Sim : Towards Effective Pre-training and Transfer from Synthetic Datacited by 26
  4. 2021Off-Policy Reinforcement Learning with Delayed Rewardscited by 25
  5. 2022Phasic Self-Imitative Reduction for Sparse-Reward Goal-Conditioned Reinforcement Learningcited by 16
  6. 2023Imitating Graph-Based Planning with Goal-Conditioned Policiescited by 13
  7. 2025Learn the Ropes, Then Trust the Wins: Self-imitation with Progressive Exploration for Agentic Reinforcement Learningcited by 10
  8. 2021Distilling Motion Planner Augmented Policies into Visual Control Policies for Robot Manipulationcited by 8
  9. 2021Optimistic Reinforcement Learning by Forward Kullback-Leibler Divergence Optimizationcited by 6
  10. 2022Evolutionary Action Selection for Gradient-based Policy Learningcited by 5
  11. 2023Visual Hindsight Self-Imitation Learning for Interactive Navigationcited by 5
  12. 2021Learning from demonstrations with SACR2: Soft Actor-Critic with Reward Relabelingcited by 4
13 more
  1. 2022Reinforcement learning with Demonstrations from Mismatched Task under Sparse Rewardcited by 4
  2. 2023Reinforcement Learning in Robotic Motion Planning by Combined Experience-based Planning and Self-Imitation Learningcited by 4
  3. 2022Emerging Threats in Deep Learning-Based Autonomous Driving: A Comprehensive Surveycited by 3
  4. 2022Towards Improving Exploration in Self-Imitation Learning using Intrinsic Motivationcited by 3
  5. 2021Co-Imitation Learning without Expert Demonstrationcited by 2
  6. 2021Shortest-Path Constrained Reinforcement Learning for Sparse Reward Taskscited by 2
  7. 2022Robust Action Gap Increasing with Clipped Advantage Learningcited by 1
  8. 2021Dual Behavior Regularized Reinforcement Learningcited by 0
  9. 2022A Fully Controllable Agent in the Path Planning using Goal-Conditioned Reinforcement Learningcited by 0
  10. 2022A Parametric Class of Approximate Gradient Updates for Policy Optimizationcited by 0
  11. 2024SIL-RRT*: Learning Sampling Distribution through Self Imitation Learningcited by 0
  12. 2026Match or Replay: Self Imitating Proximal Policy Optimizationcited by 0
  13. 2026Temporal Self-Imitation Learningcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024PLANRL: A Motion Planning and Imitation Learning Framework to Bootstrap Reinforcement Learning
  2. 2025Self-Augmented Robot Trajectory: Efficient Imitation Learning via Safe Self-augmentation with Demonstrator-annotated Precision
  3. 2026Self-Imitated Diffusion Policy for Efficient and Robust Visual Navigation
  4. 2019Relay Policy Learning: Solving Long-Horizon Tasks via Imitation and Reinforcement Learning
  5. 2024MILES: Making Imitation Learning Easy with Self-Supervision
  6. 2024Instant Policy: In-Context Imitation Learning via Graph Diffusion
  7. 2022Divide & Conquer Imitation Learning
  8. 2022Self-Imitation Learning from Demonstrations