# Self-Imitation Advantage Learning

arXiv: https://arxiv.org/abs/2012.11989 · HTML: https://mlsanity.com/paper/2012.11989/

## Key facts

- Cited by 15 papers in the map (data through 14 Oct 2026).
- Placed in Imitation learning methods, one of 13 directions in Offline RL and imitation learning from data.
- First posted on arXiv 22 Dec 2020.

## Placed in

- Offline RL and imitation learning from data / [Imitation learning methods](https://mlsanity.com/offlinerl/imitation-learning-methods/)

## Built on by (most cited first, 15 of 15)

- [Hit and Lead Discovery with Explorative RL and Fragment-based Molecule Generation](https://mlsanity.com/paper/2110.01219/) (2021)
- [Exploration in Deep Reinforcement Learning: From Single-Agent to Multiagent Domain](https://mlsanity.com/paper/2109.06668/) (2021)
- [Adversarially Guided Actor-Critic](https://mlsanity.com/paper/2102.04376/) (2021)
- [Learn the Ropes, Then Trust the Wins: Self-imitation with Progressive Exploration for Agentic Reinforcement Learning](https://mlsanity.com/paper/2509.22601/) (2025)
- [ReLU to the Rescue: Improve Your On-Policy Actor-Critic with Positive Advantages](https://mlsanity.com/paper/2306.01460/) (2023)
- [Lazy-MDPs: Towards Interpretable Reinforcement Learning by Learning When to Act](https://mlsanity.com/paper/2203.08542/) (2022)
- [A Closer Look at Advantage-Filtered Behavioral Cloning in High-Noise Datasets](https://mlsanity.com/paper/2110.04698/) (2021)
- [Goal-conditioned Offline Reinforcement Learning through State Space Partitioning](https://mlsanity.com/paper/2303.09367/) (2023)
- [STIR2: Reward Relabelling for combined Reinforcement and Imitation Learning on sparse-reward tasks](https://mlsanity.com/paper/2201.03834/) (2022)
- [Smoothing Advantage Learning](https://mlsanity.com/paper/2203.10445/) (2022)
- [Robust Action Gap Increasing with Clipped Advantage Learning](https://mlsanity.com/paper/2203.11677/) (2022)
- [Simultaneous Double Q-learning with Conservative Advantage Learning for Actor-Critic Methods](https://mlsanity.com/paper/2205.03819/) (2022)
- [Enforcing KL Regularization in General Tsallis Entropy Reinforcement Learning via Advantage Learning](https://mlsanity.com/paper/2205.07885/) (2022)
- [Match or Replay: Self Imitating Proximal Policy Optimization](https://mlsanity.com/paper/2603.27515/) (2026)
- [Optimizing Neurorobot Policy under Limited Demonstration Data through Preference Regret](https://mlsanity.com/paper/2604.03523/) (2026)

## Abstract (arXiv)

Self-imitation learning is a Reinforcement Learning (RL) method that encourages actions whose returns were higher than expected, which helps in hard exploration and sparse reward problems. It was shown to improve the performance of on-policy actor-critic methods in several discrete control tasks. Nevertheless, applying self-imitation to the mostly action-value based off-policy RL methods is not straightforward. We propose SAIL, a novel generalization of self-imitation learning for off-policy RL, based on a modification of the Bellman optimality operator that we connect to Advantage Learning. Crucially, our method mitigates the problem of stale returns by choosing the most optimistic return estimate between the observed return and the current action-value for self-imitation. We demonstrate the empirical effectiveness of SAIL on the Arcade Learning Environment, with a focus on hard exploration games.