mlsanityData to 7 Oct 2026

Relay Hindsight Experience Replay: Self-Guided Continual Reinforcement Learning for Sequential Object Manipulation Tasks with Sparse Rewards

arXiv 2208.00843 ↗First posted 1 Aug 2022cs.RO cs.AIcited by 10

Not on the map. Why?

Abstract (arXiv)

Exploration with sparse rewards remains a challenging research problem in reinforcement learning (RL). Especially for sequential object manipulation tasks, the RL agent always receives negative rewards until completing all sub-tasks, which results in low exploration efficiency. To solve these tasks efficiently, we propose a novel self-guided continual RL framework, RelayHER (RHER). RHER first decomposes a sequential task into new sub-tasks with increasing complexity and ensures that the simplest sub-task can be learned quickly by utilizing Hindsight Experience Replay (HER). Secondly, we design a multi-goal & multi-task network to learn these sub-tasks simultaneously. Finally, we propose a Self-Guided Exploration Strategy (SGES). With SGES, the learned sub-task policy will guide the agent to the states that are helpful to learn more complex sub-task with HER. By this self-guided exploration and relay policy learning, RHER can solve these sequential tasks efficiently stage by stage. The experimental results show that RHER significantly outperforms vanilla-HER in sample-efficiency on five singleobject and five complex multi-object manipulation tasks (e.g., Push, Insert, ObstaclePush, Stack, TStack, etc.). The proposed RHER has also been applied to learn a contact-rich push task on a physical robot from scratch, and the success rate reached 10/10 with only 250 episodes.

Key facts. Cited by 10 papers in the map (data through 7 Oct 2026). Builds on 28 of its 47 references that are in the map. First posted on arXiv 1 Aug 2022.

Builds on

28 of 47 references.

References · 28

  1. 2017Proximal Policy Optimization Algorithmscited by 14,441
  2. 2015Continuous control with deep reinforcement learningcited by 3,279
  3. 2018Addressing Function Approximation Error in Actor-Critic Methodscited by 2,302
  4. 2016OpenAI Gymcited by 1,684
  5. 2019Meta-World: A Benchmark and Evaluation for Multi-Task and Meta Reinforcement Learningcited by 1,168
  6. 2017Hindsight Experience Replaycited by 1,096
  7. 2017Memory Aware Synapses: Learning what (not) to forgetcited by 1,094
  8. 2018Efficient Lifelong Learning with A-GEMcited by 1,044
  9. 2018Learning to Learn without Forgetting by Maximizing Transfer and Minimizing Interferencecited by 511
  10. 2016Deep Reinforcement Learning for Robotic Manipulation with Asynchronous Off-Policy Updatescited by 342
  11. 2017Overcoming Exploration in Reinforcement Learning with Demonstrationscited by 322
  12. 2019Continual learning with hypernetworkscited by 267
16 more
  1. 2018Multi-Goal Reinforcement Learning: Challenging Robotics Environments and Request for Researchcited by 256
  2. 2021MT-Opt: Continuous Multi-Task Robotic Reinforcement Learning at Scalecited by 226
  3. 2018Residual Policy Learningcited by 146
  4. 2017Routing Networks: Adaptive Selection of Non-linear Functions for Multi-Task Learningcited by 125
  5. 2018Measuring and regularizing networks in function spacecited by 97
  6. 2019Deep Reinforcement Learning for Industrial Insertion Tasks with Visual Inputs and Natural Rewardscited by 58
  7. 2018Modular meta-learningcited by 51
  8. 2020Continual Model-Based Reinforcement Learning with Hypernetworkscited by 41
  9. 2021Lifelong Robotic Reinforcement Learning by Retaining Experiencescited by 34
  10. 2018Energy-Based Hindsight Experience Prioritizationcited by 20
  11. 2020Curriculum Learning with Hindsight Experience Replay for Sequential Object Manipulation Taskscited by 10
  12. 2021Imaginary Hindsight Experience Replay: Curious Model-based Learning for Sparse Reward Taskscited by 8
  13. 2021An Open-Source Multi-Goal Reinforcement Learning Environment for Robotic Manipulation with Pybulletcited by 5
  14. 2019Bias-Reduced Hindsight Experience Replay with Virtual Goal Prioritizationcited by 4
  15. 2020ACDER: Augmented Curiosity-Driven Experience Replaycited by 4
  16. 2022Abstract Demonstrations and Adaptive Exploration for Efficient and Stable Multi-step Sparse Reward Reinforcement Learningcited by 1

Built on by

Cited by year

Citing papers per year: 2022: 1; 2023: 2; 2024: 3; 2025: 4.

10 papers.

  1. 2025Act2Goal: From World Model To General Goal-conditioned Policycited by 18
  2. 2025A Survey of Continual Reinforcement Learningcited by 16
  3. 2024Unveiling the Anomalies in an Ever-Changing World: A Benchmark for Pixel-Level Anomaly Detection in Continual Learningcited by 13
  4. 2022Continual Learning Approaches for Anomaly Detectioncited by 9
  5. 2024Efficient Diversity-based Experience Replay for Deep Reinforcement Learningcited by 7
  6. 2025Sample Efficient Experience Replay in Non-stationary Environmentscited by 2
  7. 2024MENTOR: Guiding Hierarchical Reinforcement Learning with Human Feedback and Dynamic Distance Constraintcited by 1
  8. 2025Dynamic Object Masks as Goal Representations for Visual Goal-Conditioned Reinforcement Learningcited by 1
  9. 2023MRHER: Model-based Relay Hindsight Experience Replay for Sequential Object Manipulation Tasks with Sparse Rewardscited by 0
  10. 2023Tactile Active Inference Reinforcement Learning for Efficient Robotic Manipulation Skill Acquisitioncited by 0