mlsanityData to 7 Oct 2026

Hindsight policy gradients

arXiv 1711.06006 ↗First posted 16 Nov 2017cs.LG cs.AI cs.NEcited by 36

Not on the map. Why?

Abstract (arXiv)

A reinforcement learning agent that needs to pursue different goals across episodes requires a goal-conditional policy. In addition to their potential to generalize desirable behavior to unseen goals, such policies may also enable higher-level planning based on subgoals. In sparse-reward environments, the capacity to exploit information about the degree to which an arbitrary goal has been achieved while another goal was intended appears crucial to enable sample efficient learning. However, reinforcement learning agents have only recently been endowed with such capacity for hindsight. In this paper, we demonstrate how hindsight can be introduced to policy gradient methods, generalizing this idea to a broad class of successful algorithms. Our experiments on a diverse selection of sparse-reward environments show that hindsight leads to a remarkable increase in sample efficiency.

Key facts. Cited by 36 papers in the map (data through 7 Oct 2026). First posted on arXiv 16 Nov 2017.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2020: 2; 2021: 10; 2022: 6; 2023: 8; 2024: 3; 2025: 4; 2026: 3.

36 papers.

Sort:

By area:

  1. 2021Offline Reinforcement Learning as One Big Sequence Modeling Problemcited by 636
  2. 2023Learning Interactive Real-World Simulatorscited by 440
  3. 2022Online Decision Transformercited by 194
  4. 2021Learning One Representation to Optimize All Rewardscited by 110
  5. 2021Exploration in Deep Reinforcement Learning: From Single-Agent to Multiagent Domaincited by 57
  6. 2021MURAL: Meta-Learning Uncertainty-Aware Rewards for Outcome-Driven Reinforcement Learningcited by 24
  7. 2022Generalization with Lossy Affordances: Leveraging Broad Offline Data for Learning Visuomotor Taskscited by 24
  8. 2020Scalable Multi-Task Imitation Learning with Autonomous Improvementcited by 21
  9. 2022Hypernetworks for Zero-shot Transfer in Reinforcement Learningcited by 18
  10. 2020Episodic Self-Imitation Learning with Hindsightcited by 10
  11. 2023Would I have gotten that reward? Long-term credit assignment by counterfactual contribution analysiscited by 10
  12. 2023Guided Data Augmentation for Offline Reinforcement Learning and Imitation Learningcited by 10
24 more
  1. 2021DisCo RL: Distribution-Conditioned Reinforcement Learning for General-Purpose Policiescited by 9
  2. 2022Goal-Conditioned Generators of Deep Policiescited by 9
  3. 2021Hierarchical Reinforcement Learning for Air Combat at DARPA's AlphaDogfight Trialscited by 8
  4. 2021Hindsight Task Relabelling: Experience Replay for Sparse Reward Meta-RLcited by 8
  5. 2021Unbiased Methods for Multi-Goal Reinforcement Learningcited by 7
  6. 2021Hindsight Goal Ranking on Replay Buffer for Sparse Reward Environmentcited by 7
  7. 2023Skill Decision Transformercited by 7
  8. 2024Directed Exploration in Reinforcement Learning from Linear Temporal Logiccited by 7
  9. 2021Diversity-based Trajectory and Goal Selection with Hindsight Experience Replaycited by 6
  10. 2022Hidden Agenda: a Social Deduction Game with Diverse Learned Equilibriacited by 6
  11. 2023Understanding when Dynamics-Invariant Data Augmentations Benefit Model-Free Reinforcement Learning Updatescited by 6
  12. 2022Open-Ended Reinforcement Learning with Neural Reward Functionscited by 5
  13. 2024Offline Goal-Conditioned Reinforcement Learning for Safety-Critical Tasks with Recovery Policycited by 5
  14. 2024Hindsight Experience Replay Accelerates Proximal Policy Optimizationcited by 2
  15. 2025BaNEL: Exploration Posteriors for Generative Modeling Using Only Negative Rewardscited by 2
  16. 2026RLFTSim: Realistic and Controllable Multi-Agent Traffic Simulation via Reinforcement Learning Fine-Tuningcited by 2
  17. 2023Hindsight States: Blending Sim and Real Task Elements for Efficient Reinforcement Learningcited by 1
  18. 2023Goal-Conditioned Supervised Learning with Sub-Goal Predictioncited by 1
  19. 2023Deep Generative Models for Decision-Making and Controlcited by 1
  20. 2025Hierarchical Reinforcement Learning in Multi-Goal Spatial Navigation with Autonomous Mobile Robotscited by 1
  21. 2025Autonomous Learning From Success and Failure: Goal-Conditioned Supervised Learning with Negative Feedbackcited by 0
  22. 2025HRM-Agent: Training a recurrent reasoning model in dynamic environments using reinforcement learningcited by 0
  23. 2026Hindsight-Anchored Policy Optimization: Learning Through Hindsight with Thompson Sampling-Inspired Adaptive Gatingcited by 0
  24. 2026Learning More from Less: Reinforcement Learning from Hindsightcited by 0