mlsanityData to 14 Oct 2026

Bias-reduced Multi-step Hindsight Experience Replay for Efficient Multi-goal Reinforcement Learning

arXiv 2102.12962 ↗First posted 25 Feb 2021cs.LG cs.AI cs.ROcited by 4

Not on the map. Why?

Abstract (arXiv)

Multi-goal reinforcement learning is widely applied in planning and robot manipulation. Two main challenges in multi-goal reinforcement learning are sparse rewards and sample inefficiency. Hindsight Experience Replay (HER) aims to tackle the two challenges via goal relabeling. However, HER-related works still need millions of samples and a huge computation. In this paper, we propose Multi-step Hindsight Experience Replay (MHER), incorporating multi-step relabeled returns based on n-step relabeling to improve sample efficiency. Despite the advantages of n-step relabeling, we theoretically and experimentally prove the off-policy n-step bias introduced by n-step relabeling may lead to poor performance in many environments. To address the above issue, two bias-reduced MHER algorithms, MHER(λ) and Model-based MHER (MMHER) are presented. MHER(λ) exploits the λ return while MMHER benefits from model-based value expansions. Experimental results on numerous multi-goal robotic tasks show that our solutions can successfully alleviate off-policy n-step bias and achieve significantly higher sample efficiency than HER and Curriculum-guided HER with little additional computation beyond HER.

Key facts. Cited by 4 papers in the map (data through 14 Oct 2026). Builds on 14 of its 31 references that are in the map. First posted on arXiv 25 Feb 2021.

Builds on

14 of 31 references.

References · 14

  1. 2018Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actorcited by 4,885
  2. 2015Continuous control with deep reinforcement learningcited by 3,407
  3. 2019Meta-World: A Benchmark and Evaluation for Multi-Task and Meta Reinforcement Learningcited by 1,204
  4. 2017Hindsight Experience Replaycited by 1,136
  5. 2017Rainbow: Combining Improvements in Deep Reinforcement Learningcited by 786
  6. 2019When to Trust Your Model: Model-Based Policy Optimizationcited by 773
  7. 2017Neural Network Dynamics for Model-Based Deep Reinforcement Learning with Model-Free Fine-Tuningcited by 297
  8. 2016Safe and Efficient Off-Policy Reinforcement Learningcited by 264
  9. 2018Multi-Goal Reinforcement Learning: Challenging Robotics Environments and Request for Researchcited by 260
  10. 2018Distributed Distributional Deterministic Policy Gradientscited by 199
  11. 2018Model-Based Value Estimation for Efficient Model-Free Reinforcement Learningcited by 156
  12. 2019Maximum Entropy-Regularized Multi-Goal Reinforcement Learningcited by 40
2 more
  1. 2015Towards Vision-Based Deep Reinforcement Learning for Robotic Motion Controlcited by 33
  2. 2018Energy-Based Hindsight Experience Prioritizationcited by 21

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2021: 1; 2022: 1; 2023: 1; 2024: 0; 2025: 0; 2026: 1.

4 papers.

  1. 2021Conservative Data Sharing for Multi-Task Offline Reinforcement Learningcited by 64
  2. 2022USHER: Unbiased Sampling for Hindsight Experience Replaycited by 6
  3. 2026Debiased Model-based Representations for Sample-efficient Continuous Controlcited by 1
  4. 2023Bias Resilient Multi-Step Off-Policy Goal-Conditioned Reinforcement Learningcited by 0