mlsanityData to 7 Oct 2026

Hindsight Goal Ranking on Replay Buffer for Sparse Reward Environment

arXiv 2110.15043 ↗First posted 28 Oct 2021cs.LGcited by 7

Not on the map. Why?

Abstract (arXiv)

This paper proposes a method for prioritizing the replay experience referred to as Hindsight Goal Ranking (HGR) in overcoming the limitation of Hindsight Experience Replay (HER) that generates hindsight goals based on uniform sampling. HGR samples with higher probability on the states visited in an episode with larger temporal difference (TD) error, which is considered as a proxy measure of the amount which the RL agent can learn from an experience. The actual sampling for large TD error is performed in two steps: first, an episode is sampled from the relay buffer according to the average TD error of its experiences, and then, for the sampled episode, the hindsight goal leading to larger TD error is sampled with higher probability from future visited states. The proposed method combined with Deep Deterministic Policy Gradient (DDPG), an off-policy model-free actor-critic algorithm, accelerates learning significantly faster than that without any prioritization on four challenging simulated robotic manipulation tasks. The empirical results show that HGR uses samples more efficiently than previous methods across all tasks.

Key facts. Cited by 7 papers in the map (data through 7 Oct 2026). Builds on 29 of its 40 references that are in the map. First posted on arXiv 28 Oct 2021.

Builds on

29 of 40 references.

References · 29

  1. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  2. 2018Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actorcited by 4,635
  3. 2015Continuous control with deep reinforcement learningcited by 3,279
  4. 2013Playing Atari with Deep Reinforcement Learningcited by 3,211
  5. 2018Addressing Function Approximation Error in Actor-Critic Methodscited by 2,302
  6. 2015Deep Reinforcement Learning with Double Q-learningcited by 1,704
  7. 2016OpenAI Gymcited by 1,684
  8. 2016Generative Adversarial Imitation Learningcited by 1,576
  9. 2017Hindsight Experience Replaycited by 1,096
  10. 2015Prioritized Experience Replaycited by 1,070
  11. 2015End-to-End Training of Deep Visuomotor Policiescited by 847
  12. 2015Dueling Network Architectures for Deep Reinforcement Learningcited by 769
17 more
  1. 2018QT-Opt: Scalable Deep Reinforcement Learning for Vision-Based Robotic Manipulationcited by 730
  2. 2017Mastering Chess and Shogi by Self-Play with a General Reinforcement Learning Algorithmcited by 723
  3. 2016Benchmarking Deep Reinforcement Learning for Continuous Controlcited by 361
  4. 2017Overcoming Exploration in Reinforcement Learning with Demonstrationscited by 322
  5. 2018Multi-Goal Reinforcement Learning: Challenging Robotics Environments and Request for Researchcited by 256
  6. 2017Automatic Goal Generation for Reinforcement Learning Agentscited by 252
  7. 2018Visual Reinforcement Learning with Imagined Goalscited by 248
  8. 2017Reverse Curriculum Generation for Reinforcement Learningcited by 185
  9. 2019Skew-Fit: State-Covering Self-Supervised Reinforcement Learningcited by 171
  10. 2019Goal-conditioned Imitation Learningcited by 159
  11. 2017Intrinsically Motivated Goal Exploration Processes with Automatic Curriculum Learningcited by 79
  12. 2018CURIOUS: Intrinsically Motivated Modular Multi-Goal Reinforcement Learningcited by 78
  13. 2020Rewriting History with Inverse RL: Hindsight Inference for Policy Improvementcited by 59
  14. 2020Generalized Hindsight for Reinforcement Learningcited by 51
  15. 2019Maximum Entropy-Regularized Multi-Goal Reinforcement Learningcited by 39
  16. 2017Hindsight policy gradientscited by 36
  17. 2018Energy-Based Hindsight Experience Prioritizationcited by 20

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 2; 2025: 4; 2026: 1.

7 papers.

  1. 2024Predictive Coding for Decision Transformercited by 6
  2. 2025Policy Learning from Large Vision-Language Model Feedback without Reward Modelingcited by 4
  3. 2025Sample Efficient Reinforcement Learning via Large Vision Language Model Distillationcited by 3
  4. 2025Reward Generation via Large Vision-Language Model in Offline Reinforcement Learningcited by 2
  5. 2024On the Perturbed States for Transformed Input-robust Reinforcement Learningcited by 0
  6. 2025SD2AIL: Adversarial Imitation Learning from Synthetic Demonstrations via Diffusion Modelscited by 0
  7. 2026VLM-Guided Experience Replaycited by 0