# Self-Imitation Learning for Robot Tasks with Sparse and Delayed Rewards

arXiv: https://arxiv.org/abs/2010.06962 · HTML: https://mlsanity.com/paper/2010.06962/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- Placed in Imitation learning methods, one of 13 directions in Offline RL and imitation learning from data.
- Builds on 12 of its 22 references that are in the map.
- First posted on arXiv 14 Oct 2020.

## Placed in

- Offline RL and imitation learning from data / [Imitation learning methods](https://mlsanity.com/offlinerl/imitation-learning-methods/)

## Builds on (most cited first, 12 of 12)

- [OpenAI Gym](https://mlsanity.com/paper/1606.01540/) (2016)
- [Curiosity-driven Exploration by Self-supervised Prediction](https://mlsanity.com/paper/1705.05363/) (2017)
- [Hindsight Experience Replay](https://mlsanity.com/paper/1707.01495/) (2017)
- [Exploration by Random Network Distillation](https://mlsanity.com/paper/1810.12894/) (2018)
- [Learning agile and dynamic motor skills for legged robots](https://mlsanity.com/paper/1901.08652/) (2019)
- [Benchmarking Deep Reinforcement Learning for Continuous Control](https://mlsanity.com/paper/1604.06778/) (2016)
- [The Ingredients of Real-World Robotic Reinforcement Learning](https://mlsanity.com/paper/2004.12570/) (2020)
- [Dexterous Manipulation with Deep Reinforcement Learning: Efficient, General, and Low-Cost](https://mlsanity.com/paper/1810.06045/) (2018)
- [Learning Self-Imitating Diverse Policies](https://mlsanity.com/paper/1805.10309/) (2018)
- [A Large-Scale Study on Regularization and Normalization in GANs](https://mlsanity.com/paper/1807.04720/) (2018)
- [Scalable Multi-Task Imitation Learning with Autonomous Improvement](https://mlsanity.com/paper/2003.02636/) (2020)
- [Learning Hierarchical Control for Robust In-Hand Manipulation](https://mlsanity.com/paper/1910.10985/) (2019)

## Built on by (most cited first, 4 of 4)

- [A Survey of Temporal Credit Assignment in Deep Reinforcement Learning](https://mlsanity.com/paper/2312.01072/) (2023)
- [Attention-Based Reward Shaping for Sparse and Delayed Rewards](https://mlsanity.com/paper/2505.10802/) (2025)
- [Towards a Reward-Free Reinforcement Learning Framework for Vehicle Control](https://mlsanity.com/paper/2502.15262/) (2025)
- [Temporal Self-Imitation Learning](https://mlsanity.com/paper/2606.19752/) (2026)

## Abstract (arXiv)

The application of reinforcement learning (RL) in robotic control is still limited in the environments with sparse and delayed rewards. In this paper, we propose a practical self-imitation learning method named Self-Imitation Learning with Constant Reward (SILCR). Instead of requiring hand-defined immediate rewards from environments, our method assigns the immediate rewards at each timestep with constant values according to their final episodic rewards. In this way, even if the dense rewards from environments are unavailable, every action taken by the agents would be guided properly. We demonstrate the effectiveness of our method in some challenging continuous robotics control tasks in MuJoCo simulation and the results show that our method significantly outperforms the alternative methods in tasks with sparse and delayed rewards. Even compared with alternatives with dense rewards available, our method achieves competitive performance. The ablation experiments also show the stability and reproducibility of our method.