mlsanityData to 14 Oct 2026

Self-Imitation Learning from Demonstrations

arXiv 2203.10905 ↗First posted 21 Mar 2022cs.LGcited by 4

On the map: Offline RL and imitation learning from data → Imitation learning methods

Abstract (arXiv)

Despite the numerous breakthroughs achieved with Reinforcement Learning (RL), solving environments with sparse rewards remains a challenging task that requires sophisticated exploration. Learning from Demonstrations (LfD) remedies this issue by guiding the agent's exploration towards states experienced by an expert. Naturally, the benefits of this approach hinge on the quality of demonstrations, which are rarely optimal in realistic scenarios. Modern LfD algorithms require meticulous tuning of hyperparameters that control the influence of demonstrations and, as we show in the paper, struggle with learning from suboptimal demonstrations. To address these issues, we extend Self-Imitation Learning (SIL), a recent RL algorithm that exploits the agent's past good experience, to the LfD setup by initializing its replay buffer with demonstrations. We denote our algorithm as SIL from Demonstrations (SILfD). We empirically show that SILfD can learn from demonstrations that are noisy or far from optimal and can automatically adjust the influence of demonstrations throughout the training without additional hyperparameters or handcrafted schedules. We also find SILfD superior to the existing state-of-the-art LfD algorithms in sparse environments, especially when demonstrations are highly suboptimal.

Key facts. Cited by 4 papers in the map (data through 14 Oct 2026). Placed in Imitation learning methods, one of 13 directions in Offline RL and imitation learning from data. Builds on 35 of its 52 references that are in the map. First posted on arXiv 21 Mar 2022.

Builds on

35 of 52 references.

Closest ancestors · 7

Same direction, newest first

  1. 2021Counter-Strike Deathmatch with Large-Scale Behavioural Cloning9 Apr
  2. 2020Adversarial Soft Advantage Fitting: Imitation Learning without Policy Optimization23 Jun
  3. 2020Self-Imitation Learning via Generalized Lower Bound Q-learning12 Jun
  4. 2019Reinforced Imitation in Heterogeneous Action Space6 Apr
  5. 2018Self-Imitation Learning14 Jun
  6. 2018Fast Policy Learning through Imitation and Reinforcement26 May
  7. 2018Reinforcement Learning from Imperfect Demonstrations14 Feb

Other references · 28

  1. 2014Adam: A Method for Stochastic Optimizationcited by 37,240
  2. 2017Proximal Policy Optimization Algorithmscited by 15,215
  3. 2015Trust Region Policy Optimizationcited by 2,580
  4. 2016Asynchronous Methods for Deep Reinforcement Learningcited by 2,258
  5. 2015High-Dimensional Continuous Control Using Generalized Advantage Estimationcited by 2,026
  6. 2015Deep Reinforcement Learning with Double Q-learningcited by 1,745
  7. 2016Generative Adversarial Imitation Learningcited by 1,597
  8. 2021Decision Transformer: Reinforcement Learning via Sequence Modelingcited by 1,546
  9. 2019Dota 2 with Large Scale Deep Reinforcement Learningcited by 1,103
  10. 2015Prioritized Experience Replaycited by 1,099
  11. 2018DeepMind Control Suitecited by 956
  12. 2015End-to-End Training of Deep Visuomotor Policiescited by 875
16 more
  1. 2015Dueling Network Architectures for Deep Reinforcement Learningcited by 785
  2. 2017Learning Complex Dexterous Manipulation with Deep Reinforcement Learning and Demonstrationscited by 715
  3. 2016Deep Exploration via Bootstrapped DQNcited by 527
  4. 2016Deep Reinforcement Learning for Robotic Manipulation with Asynchronous Off-Policy Updatescited by 353
  5. 2017Overcoming Exploration in Reinforcement Learning with Demonstrationscited by 340
  6. 2017Leveraging Demonstrations for Deep Reinforcement Learning on Robotics Problems with Sparse Rewardscited by 319
  7. 2018Human-level performance in first-person multiplayer games with population-based deep reinforcement learningcited by 264
  8. 2017DeepStack: Expert-Level Artificial Intelligence in No-Limit Pokercited by 233
  9. 2018Randomized Prior Functions for Deep Reinforcement Learningcited by 219
  10. 2018Reinforcement and Imitation Learning for Diverse Visuomotor Skillscited by 126
  11. 2020Combining Deep Reinforcement Learning and Search for Imperfect-Information Gamescited by 114
  12. 2019Making Efficient Use of Demonstrations to Solve Hard Exploration Problemscited by 45
  13. 2018Pommerman: A Multi-Agent Playgroundcited by 30
  14. 2019Reinforcement Learning from Imperfect Demonstrations under Soft Expert Guidancecited by 25
  15. 2020Sample Efficient Reinforcement Learning through Learning from Demonstrations in Minecraftcited by 15
  16. 2020Show me the Way: Intrinsic Motivation from Demonstrationscited by 7

Built on by

Cited by year

Citing papers per year: 2022: 2; 2023: 1; 2024: 0; 2025: 1.

4 papers.

  1. 2025Learn the Ropes, Then Trust the Wins: Self-imitation with Progressive Exploration for Agentic Reinforcement Learningcited by 11
  2. 2023Visual Hindsight Self-Imitation Learning for Interactive Navigationcited by 6
  3. 2022Towards Improving Exploration in Self-Imitation Learning using Intrinsic Motivationcited by 3
  4. 2022Accelerating Self-Imitation Learning from Demonstrations via Policy Constraints and Q-Ensemblecited by 1

Similar, not cited

Close in wording, no citation either way

  1. 2019Watch, Try, Learn: Meta-Learning from Demonstrations and Reward
  2. 2019SQIL: Imitation Learning via Reinforcement Learning with Sparse Rewards
  3. 2022Reinforcement learning with Demonstrations from Mismatched Task under Sparse Reward
  4. 2022STIR2: Reward Relabelling for combined Reinforcement and Imitation Learning on sparse-reward tasks
  5. 2021Learning from Ambiguous Demonstrations with Self-Explanation Guided Reinforcement Learning
  6. 2020Learning from Suboptimal Demonstration via Self-Supervised Reward Regression
  7. 2021Learning from demonstrations with SACR2: Soft Actor-Critic with Reward Relabeling
  8. 2019A Divergence Minimization Perspective on Imitation Learning Methods