mlsanityData to 7 Oct 2026

Self-Imitation Learning

arXiv 1806.05635 ↗First posted 14 Jun 2018cs.LG cs.AI stat.MLcited by 135

On the map: Offline RL and imitation learning from data → Imitation learning methods

Abstract (arXiv)

This paper proposes Self-Imitation Learning (SIL), a simple off-policy actor-critic algorithm that learns to reproduce the agent's past good decisions. This algorithm is designed to verify our hypothesis that exploiting past good experiences can indirectly drive deep exploration. Our empirical results show that SIL significantly improves advantage actor-critic (A2C) on several hard exploration Atari games and is competitive to the state-of-the-art count-based exploration methods. We also show that SIL improves proximal policy optimization (PPO) on MuJoCo tasks.

Key facts. Cited by 135 papers in the map (data through 7 Oct 2026). Placed in Imitation learning methods, one of 13 directions in Offline RL and imitation learning from data. Builds on 24 of its 42 references that are in the map. First posted on arXiv 14 Jun 2018.

Builds on

24 of 42 references.

Closest ancestors · 1

Same direction, newest first

  1. 2018Reinforcement Learning from Imperfect Demonstrations14 Feb

Other references · 23

  1. 2017Proximal Policy Optimization Algorithmscited by 14,441
  2. 2015Continuous control with deep reinforcement learningcited by 3,279
  3. 2016Asynchronous Methods for Deep Reinforcement Learningcited by 2,218
  4. 2016OpenAI Gymcited by 1,684
  5. 2015Prioritized Experience Replaycited by 1,070
  6. 2015Dueling Network Architectures for Deep Reinforcement Learningcited by 769
  7. 2017Reinforcement Learning with Deep Energy-Based Policiescited by 659
  8. 2016Unifying Count-Based Exploration and Intrinsic Motivationcited by 632
  9. 2016Deep Exploration via Bootstrapped DQNcited by 518
  10. 2017Overcoming Exploration in Reinforcement Learning with Demonstrationscited by 322
  11. 2016#Exploration: A Study of Count-Based Exploration for Deep Reinforcement Learningcited by 307
  12. 2016Safe and Efficient Off-Policy Reinforcement Learningcited by 260
11 more
  1. 2017Count-Based Exploration with Neural Density Modelscited by 255
  2. 2016Sample Efficient Actor-Critic with Experience Replaycited by 185
  3. 2017Bridging the Gap Between Value and Policy Based Reinforcement Learningcited by 178
  4. 2017Equivalence Between Policy Gradients and Soft Q-Learningcited by 172
  5. 2015Incentivizing Exploration In Reinforcement Learning With Deep Predictive Modelscited by 154
  6. 2017Neural Episodic Controlcited by 148
  7. 2016Neural Symbolic Machines: Learning Semantic Parsers on Freebase with Weak Supervisioncited by 101
  8. 2016Model-Free Episodic Controlcited by 100
  9. 2018Neural Program Synthesis with Priority Queue Trainingcited by 24
  10. 2016Learning to Play in a Day: Faster Deep Reinforcement Learning by Optimality Tighteningcited by 21
  11. 2017The Reactor: A fast and sample-efficient Actor-Critic agent for Reinforcement Learningcited by 7

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2018: 2; 2019: 1; 2020: 1; 2021: 22; 2022: 25; 2023: 21; 2024: 27; 2025: 16; 2026: 20.

135 papers.

Sort:

By area:

  1. 2023Reinforced Self-Training (ReST) for Language Modelingcited by 486
  2. 2022MineDojo: Building Open-Ended Embodied Agents with Internet-Scale Knowledgecited by 469
  3. 2018Reinforced Cross-Modal Matching and Self-Supervised Imitation Learning for Vision-Language Navigationcited by 304
  4. 2022Exploration in Deep Reinforcement Learning: A Surveycited by 158
  5. 2021Actionable Models: Unsupervised Offline Reinforcement Learning of Robotic Skillscited by 125
  6. 2024xRAG: Extreme Context Compression for Retrieval-augmented Generation with One Tokencited by 123
  7. 2022Rethinking Goal-conditioned Supervised Learning and Its Connection to Offline RLcited by 89
  8. 2024SOTOPIA-π: Interactive Learning of Socially Intelligent Language Agentscited by 89
  9. 2025Subliminal Learning: Language models transmit behavioral traits via hidden signals in datacited by 89
  10. 2024Self-playing Adversarial Language Game Enhances LLM Reasoningcited by 80
  11. 2021Asymmetric self-play for automatic goal discovery in robotic manipulationcited by 71
  12. 2024Policy Agnostic RL: Offline RL and Online RL Fine-Tuning of Any Class and Backbonecited by 71
38 more
  1. 2021Hit and Lead Discovery with Explorative RL and Fragment-based Molecule Generationcited by 68
  2. 2023Auto MC-Reward: Automated Dense Reward Design with Large Language Models for Minecraftcited by 60
  3. 2021Exploration in Deep Reinforcement Learning: From Single-Agent to Multiagent Domaincited by 57
  4. 2021Imitation Learning: Progress, Taxonomies and Challengescited by 56
  5. 2021Adversarially Guided Actor-Criticcited by 47
  6. 2022Learning Generalizable Dexterous Manipulation from Human Grasp Affordancecited by 46
  7. 2023Towards A Unified Agent with Foundation Modelscited by 46
  8. 2025Residual Off-Policy RL for Finetuning Behavior Cloning Policiescited by 42
  9. 2023Contrastive Difference Predictive Codingcited by 35
  10. 2025Flow-Based Policy for Online Reinforcement Learningcited by 33
  11. 2021Rank the Episodes: A Simple Approach for Exploration in Procedurally-Generated Environmentscited by 28
  12. 2021JueWu-MC: Playing Minecraft with Sample-efficient Hierarchical Reinforcement Learningcited by 28
  13. 2018Generative Adversarial Self-Imitation Learningcited by 25
  14. 2021Self-Imitation Learning by Planningcited by 25
  15. 2022Multi-Stage Episodic Control for Strategic Exploration in Text Gamescited by 22
  16. 2022Leveraging Approximate Symbolic Models for Reinforcement Learning via Skill Diversitycited by 22
  17. 2022Boosting Offline Reinforcement Learning via Data Rebalancingcited by 22
  18. 2022I2D2: Inductive Knowledge Distillation with NeuroLogic and Self-Imitationcited by 21
  19. 2022Imitating Past Successes can be Very Suboptimalcited by 20
  20. 2021Revisiting Peng's Q(λ) for Modern Reinforcement Learningcited by 19
  21. 2021Solving Compositional Reinforcement Learning Problems via Task Reductioncited by 19
  22. 2021Regularized Softmax Deep Multi-Agent Q-Learningcited by 19
  23. 2021TiKick: Towards Playing Multi-agent Football Full Games from Single-agent Demonstrationscited by 18
  24. 2021Co-Adaptation of Algorithmic and Implementational Innovations in Inference-based Deep Reinforcement Learningcited by 15
  25. 2022Learning Category-Level Generalizable Object Manipulation Policy via Generative Adversarial Self-Imitation Learning from Demonstrationscited by 15
  26. 2024Continuous Control with Coarse-to-fine Reinforcement Learningcited by 15
  27. 2020Self-Imitation Learning via Generalized Lower Bound Q-learningcited by 13
  28. 2022Provably Efficient Offline Reinforcement Learning with Trajectory-Wise Rewardcited by 13
  29. 2023ReLU to the Rescue: Improve Your On-Policy Actor-Critic with Positive Advantagescited by 10
  30. 2025Learn the Ropes, Then Trust the Wins: Self-imitation with Progressive Exploration for Agentic Reinforcement Learningcited by 10
  31. 2021Simplifying Deep Reinforcement Learning via Self-Supervisioncited by 9
  32. 2022Personalized Reward Learning with Interaction-Grounded Learning (IGL)cited by 9
  33. 2024SORREL: Suboptimal-Demonstration-Guided Reinforcement Learning for Learning to Branchcited by 9
  34. 2025DEAS: DEtached value learning with Action Sequence for Scalable Offline RLcited by 9
  35. 2023Value-Informed Skill Chaining for Policy Learning of Long-Horizon Tasks with Surgical Robotcited by 8
  36. 2024Reinforcement Learning Gradients as Vitamin for Online Finetuning Decision Transformerscited by 8
  37. 2019Independent Generative Adversarial Self-Imitation Learning in Cooperative Multiagent Systemscited by 7
  38. 2022Lazy-MDPs: Towards Interpretable Reinforcement Learning by Learning When to Actcited by 7

50 most cited of 135 All 135 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2020Self-Imitation Advantage Learning
  2. 2025Beyond-Expert Performance with Limited Demonstrations: Efficient Imitation Learning with Double Exploration
  3. 2022A Ranking Game for Imitation Learning
  4. 2022Towards Improving Exploration in Self-Imitation Learning using Intrinsic Motivation
  5. 2020Episodic Self-Imitation Learning with Hindsight
  6. 2019Go-Explore: a New Approach for Hard-Exploration Problems
  7. 2025IL-SOAR : Imitation Learning with Soft Optimistic Actor cRitic
  8. 2025An Introduction to Deep Reinforcement and Imitation Learning