mlsanityData to 7 Oct 2026

Self-Imitation Advantage Learning

arXiv 2012.11989 ↗First posted 22 Dec 2020cs.LGcited by 15

On the map: Offline RL and imitation learning from data → Imitation learning methods

Abstract (arXiv)

Self-imitation learning is a Reinforcement Learning (RL) method that encourages actions whose returns were higher than expected, which helps in hard exploration and sparse reward problems. It was shown to improve the performance of on-policy actor-critic methods in several discrete control tasks. Nevertheless, applying self-imitation to the mostly action-value based off-policy RL methods is not straightforward. We propose SAIL, a novel generalization of self-imitation learning for off-policy RL, based on a modification of the Bellman optimality operator that we connect to Advantage Learning. Crucially, our method mitigates the problem of stale returns by choosing the most optimistic return estimate between the observed return and the current action-value for self-imitation. We demonstrate the empirical effectiveness of SAIL on the Arcade Learning Environment, with a focus on hard exploration games.

Key facts. Cited by 15 papers in the map (data through 7 Oct 2026). Placed in Imitation learning methods, one of 13 directions in Offline RL and imitation learning from data. First posted on arXiv 22 Dec 2020.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 4; 2022: 6; 2023: 2; 2024: 0; 2025: 1; 2026: 2.

15 papers.

Sort:

By area:

  1. 2021Hit and Lead Discovery with Explorative RL and Fragment-based Molecule Generationcited by 68
  2. 2021Exploration in Deep Reinforcement Learning: From Single-Agent to Multiagent Domaincited by 57
  3. 2021Adversarially Guided Actor-Criticcited by 47
  4. 2023ReLU to the Rescue: Improve Your On-Policy Actor-Critic with Positive Advantagescited by 10
  5. 2025Learn the Ropes, Then Trust the Wins: Self-imitation with Progressive Exploration for Agentic Reinforcement Learningcited by 10
  6. 2022Lazy-MDPs: Towards Interpretable Reinforcement Learning by Learning When to Actcited by 7
  7. 2021A Closer Look at Advantage-Filtered Behavioral Cloning in High-Noise Datasetscited by 5
  8. 2023Goal-conditioned Offline Reinforcement Learning through State Space Partitioningcited by 3
  9. 2022STIR2: Reward Relabelling for combined Reinforcement and Imitation Learning on sparse-reward taskscited by 1
  10. 2022Smoothing Advantage Learningcited by 1
  11. 2022Robust Action Gap Increasing with Clipped Advantage Learningcited by 1
  12. 2022Simultaneous Double Q-learning with Conservative Advantage Learning for Actor-Critic Methodscited by 0
3 more
  1. 2022Enforcing KL Regularization in General Tsallis Entropy Reinforcement Learning via Advantage Learningcited by 0
  2. 2026Match or Replay: Self Imitating Proximal Policy Optimizationcited by 0
  3. 2026Optimizing Neurorobot Policy under Limited Demonstration Data through Preference Regretcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2020Learning Sparse Rewarded Tasks from Sub-Optimal Demonstrations
  2. 2018Self-Imitation Learning
  3. 2019Reinforced Imitation in Heterogeneous Action Space
  4. 2022A Ranking Game for Imitation Learning
  5. 2025Enhancing Online Reinforcement Learning with Meta-Learned Objective from Offline Data
  6. 2024Align Your Intents: Offline Imitation Learning via Optimal Transport
  7. 2025Beyond-Expert Performance with Limited Demonstrations: Efficient Imitation Learning with Double Exploration
  8. 2022A Policy-Guided Imitation Approach for Offline Reinforcement Learning