mlsanityData to 14 Oct 2026

Hindsight Experience Replay Accelerates Proximal Policy Optimization

arXiv 2410.22524 ↗First posted 29 Oct 2024cs.LGcited by 2

On the map: Deep reinforcement learning methods → Deep RL algorithms and representations

Abstract (arXiv)

Hindsight experience replay (HER) accelerates off-policy reinforcement learning algorithms for environments that emit sparse rewards by modifying the goal of the episode post-hoc to be some state achieved during the episode. Because post-hoc modification of the observed goal violates the assumptions of on-policy algorithms, HER is not typically applied to on-policy algorithms. Here, we show that HER can dramatically accelerate proximal policy optimization (PPO), an on-policy reinforcement learning algorithm, when tested on a custom predator-prey environment.

Key facts. Cited by 2 papers in the map (data through 14 Oct 2026). Placed in Deep RL algorithms and representations, one of 15 directions in Deep reinforcement learning methods. Builds on 11 of its 20 references that are in the map. First posted on arXiv 29 Oct 2024.

Builds on

11 of 20 references.

References · 11

  1. 2017Proximal Policy Optimization Algorithmscited by 15,215
  2. 2018Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actorcited by 4,885
  3. 2015Continuous control with deep reinforcement learningcited by 3,407
  4. 2013Playing Atari with Deep Reinforcement Learningcited by 3,274
  5. 2018Addressing Function Approximation Error in Actor-Critic Methodscited by 2,417
  6. 2017Hindsight Experience Replaycited by 1,136
  7. 2018Multi-Goal Reinforcement Learning: Challenging Robotics Environments and Request for Researchcited by 260
  8. 2018Self-Imitation Learningcited by 138
  9. 2017Hindsight policy gradientscited by 37
  10. 2020Episodic Self-Imitation Learning with Hindsightcited by 10
  11. 2019Hindsight Trust Region Policy Optimizationcited by 2

Built on by

Cited by year

Citing papers per year: 2024: 1; 2025: 1.

2 papers.

  1. 2024Maximum Entropy Hindsight Experience Replaycited by 0
  2. 2025Efficient On-Policy Reinforcement Learning via Exploration of Sparse Parameter Spacecited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Adaptable Hindsight Experience Replay for Search-Based Learning
  2. 2026Enabling Option Learning in Sparse Rewards with Hindsight Experience Replay
  3. 2026Variance Reduction Based Experience Replay for Policy Optimization
  4. 2024Revisiting Experience Replayable Conditions
  5. 2025Variance Reduction via Resampling and Experience Replay
  6. 2025GCHR : Goal-Conditioned Hindsight Regularization for Sample-Efficient Reinforcement Learning
  7. 2025Sample Efficient Experience Replay in Non-stationary Environments
  8. 2024AdaMemento: Adaptive Memory-Assisted Policy Optimization for Reinforcement Learning