mlsanityData to 14 Oct 2026

MRHER: Model-based Relay Hindsight Experience Replay for Sequential Object Manipulation Tasks with Sparse Rewards

arXiv 2306.16061 ↗First posted 28 Jun 2023cs.RO cs.AIcited by 1

Not on the map. Why?

Abstract (arXiv)

Sparse rewards pose a significant challenge to achieving high sample efficiency in goal-conditioned reinforcement learning (RL). Specifically, in sequential manipulation tasks, the agent receives failure rewards until it successfully completes the entire manipulation task, which leads to low sample efficiency. To tackle this issue and improve sample efficiency, we propose a novel model-based RL framework called Model-based Relay Hindsight Experience Replay (MRHER). MRHER breaks down a continuous task into subtasks with increasing complexity and utilizes the previous subtask to guide the learning of the subsequent one. Instead of using Hindsight Experience Replay (HER) in every subtask, we design a new robust model-based relabeling method called Foresight relabeling (FR). FR predicts the future trajectory of the hindsight state and relabels the expected goal as a goal achieved on the virtual future trajectory. By incorporating FR, MRHER effectively captures more information from historical experiences, leading to improved sample efficiency, particularly in object-manipulation environments. Experimental results demonstrate that MRHER exhibits state-of-the-art sample efficiency in benchmark tasks, outperforming RHER by 13.79% and 14.29% in the FetchPush-v1 environment and FetchPickandPlace-v1 environment, respectively.

Key facts. Cited by 1 paper in the map (data through 14 Oct 2026). Builds on 15 of its 26 references that are in the map. First posted on arXiv 28 Jun 2023.

Builds on

15 of 26 references.

References · 15

  1. 2018Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actorcited by 4,885
  2. 2015Continuous control with deep reinforcement learningcited by 3,407
  3. 2016OpenAI Gymcited by 1,736
  4. 2017Hindsight Experience Replaycited by 1,136
  5. 2017Neural Network Dynamics for Model-Based Deep Reinforcement Learning with Model-Free Fine-Tuningcited by 297
  6. 2018Sample-Efficient Reinforcement Learning with Stochastic Ensemble Value Expansioncited by 168
  7. 2018Model-Based Value Estimation for Efficient Model-Free Reinforcement Learningcited by 156
  8. 2021Adversarial Intrinsic Motivation for Reinforcement Learningcited by 40
  9. 2020PlanGAN: Model-based Planning With Sparse Rewards and Multiple Goalscited by 21
  10. 2021MHER: Model-based Hindsight Experience Replaycited by 15
  11. 2021MapGo: Model-Assisted Policy Optimization for Goal-Oriented Taskscited by 13
  12. 2022Relay Hindsight Experience Replay: Self-Guided Continual Reinforcement Learning for Sequential Object Manipulation Tasks with Sparse Rewardscited by 11
3 more
  1. 2020Curriculum Learning with Hindsight Experience Replay for Sequential Object Manipulation Taskscited by 10
  2. 2021Imaginary Hindsight Experience Replay: Curious Model-based Learning for Sparse Reward Taskscited by 8
  3. 2019Bias-Reduced Hindsight Experience Replay with Virtual Goal Prioritizationcited by 4

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 1.

1 paper.

  1. 2026Combined Constrained Sampling and Reinforcement Learning for Robotic Manipulationcited by 3