mlsanityData to 7 Oct 2026

STIR2: Reward Relabelling for combined Reinforcement and Imitation Learning on sparse-reward tasks

arXiv 2201.03834 ↗First posted 11 Jan 2022cs.LG cs.ROcited by 1

On the map: Offline RL and imitation learning from data → Imitation learning methods

Abstract (arXiv)

In the search for more sample-efficient reinforcement-learning (RL) algorithms, a promising direction is to leverage as much external off-policy data as possible. For instance, expert demonstrations. In the past, multiple ideas have been proposed to make good use of the demonstrations added to the replay buffer, such as pretraining on demonstrations only or minimizing additional cost functions. We present a new method, able to leverage both demonstrations and episodes collected online in any sparse-reward environment with any off-policy algorithm. Our method is based on a reward bonus given to demonstrations and successful episodes (via relabeling), encouraging expert imitation and self-imitation. Our experiments focus on several robotic-manipulation tasks across two different simulation environments. We show that our method based on reward relabeling improves the performance of the base algorithm (SAC and DDPG) on these tasks. Finally, our best algorithm STIR2 (Self and Teacher Imitation by Reward Relabeling), which integrates into our method multiple improvements from previous works, is more data-efficient than all baselines.

Key facts. Cited by 1 paper in the map (data through 7 Oct 2026). Placed in Imitation learning methods, one of 13 directions in Offline RL and imitation learning from data. Builds on 38 of its 50 references that are in the map. First posted on arXiv 11 Jan 2022.

Builds on

38 of 50 references.

Closest ancestors · 5

Same direction, newest first

  1. 2021Learning from demonstrations with SACR2: Soft Actor-Critic with Reward Relabeling27 Oct
  2. 2021Implicit Behavioral Cloning1 Sep
  3. 2020Self-Imitation Advantage Learning22 Dec
  4. 2018Self-Imitation Learning14 Jun
  5. 2018Reinforcement Learning from Imperfect Demonstrations14 Feb

Other references · 33

  1. 2018Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actorcited by 4,635
  2. 2015Continuous control with deep reinforcement learningcited by 3,279
  3. 2016Generative Adversarial Imitation Learningcited by 1,576
  4. 2017Curiosity-driven Exploration by Self-supervised Predictioncited by 1,173
  5. 2019Meta-World: A Benchmark and Evaluation for Multi-Task and Meta Reinforcement Learningcited by 1,168
  6. 2017Hindsight Experience Replaycited by 1,096
  7. 2015Prioritized Experience Replaycited by 1,070
  8. 2016End to End Learning for Self-Driving Carscited by 1,000
  9. 2018Exploration by Random Network Distillationcited by 966
  10. 2017Rainbow: Combining Improvements in Deep Reinforcement Learningcited by 774
  11. 2017Learning Complex Dexterous Manipulation with Deep Reinforcement Learning and Demonstrationscited by 685
  12. 2019RLBench: The Robot Learning Benchmark & Learning Environmentcited by 669
21 more
  1. 2016Unifying Count-Based Exploration and Intrinsic Motivationcited by 632
  2. 2016Guided Cost Learning: Deep Inverse Optimal Control via Policy Optimizationcited by 400
  3. 2020Transporter Networks: Rearranging the Visual World for Robotic Manipulationcited by 342
  4. 2017Overcoming Exploration in Reinforcement Learning with Demonstrationscited by 322
  5. 2016#Exploration: A Study of Count-Based Exploration for Deep Reinforcement Learningcited by 307
  6. 2017Leveraging Demonstrations for Deep Reinforcement Learning on Robotics Problems with Sparse Rewardscited by 302
  7. 2016Deep Visual Foresight for Planning Robot Motioncited by 276
  8. 2021MT-Opt: Continuous Multi-Task Robotic Reinforcement Learning at Scalecited by 226
  9. 2015Deep Spatial Autoencoders for Visuomotor Learningcited by 149
  10. 2018Reinforcement and Imitation Learning for Diverse Visuomotor Skillscited by 124
  11. 2021Coarse-to-Fine Q-attention: Efficient Learning for Visual Robotic Manipulation via Discretisationcited by 120
  12. 2019Reward-Conditioned Policiescited by 101
  13. 2021Reset-Free Reinforcement Learning via Multi-Task Learning: Learning Dexterous Manipulation Behaviors without Human Interventioncited by 90
  14. 2017Neural Task Programming: Learning to Generalize Across Hierarchical Taskscited by 75
  15. 2016Loss is its own Reward: Self-Supervision for Reinforcement Learningcited by 72
  16. 2021Accelerating Robotic Reinforcement Learning via Parameterized Action Primitivescited by 63
  17. 2018Neural Task Graphs: Generalizing to Unseen Tasks from a Single Video Demonstrationcited by 60
  18. 2020Rewriting History with Inverse RL: Hindsight Inference for Policy Improvementcited by 59
  19. 2021GRI: General Reinforced Imitation and its Application to Vision-Based Autonomous Drivingcited by 59
  20. 2020Generalized Hindsight for Reinforcement Learningcited by 51
  21. 2019Learning Predictive Models From Observation and Interactioncited by 41

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2026: 1.

1 paper.

  1. 2026Temporal Self-Imitation Learningcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026Coherent Off-Policy Improvement of Large Behavior Models with Learned Rewards
  2. 2019Reinforced Imitation in Heterogeneous Action Space
  3. 2019SQIL: Imitation Learning via Reinforcement Learning with Sparse Rewards
  4. 2022Reinforcement learning with Demonstrations from Mismatched Task under Sparse Reward
  5. 2022Self-Imitation Learning from Demonstrations
  6. 2025Enhancing Online Reinforcement Learning with Meta-Learned Objective from Offline Data
  7. 2019Watch, Try, Learn: Meta-Learning from Demonstrations and Reward
  8. 2021Sample Efficient Imitation Learning via Reward Function Trained in Advance