# Hindsight policy gradients

arXiv: https://arxiv.org/abs/1711.06006 · HTML: https://mlsanity.com/paper/1711.06006/

## Key facts

- Cited by 37 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 16 Nov 2017.

## Built on by (most cited first, 30 of 37)

- [Offline Reinforcement Learning as One Big Sequence Modeling Problem](https://mlsanity.com/paper/2106.02039/) (2021)
- [Learning Interactive Real-World Simulators](https://mlsanity.com/paper/2310.06114/) (2023)
- [Online Decision Transformer](https://mlsanity.com/paper/2202.05607/) (2022)
- [Learning One Representation to Optimize All Rewards](https://mlsanity.com/paper/2103.07945/) (2021)
- [Exploration in Deep Reinforcement Learning: From Single-Agent to Multiagent Domain](https://mlsanity.com/paper/2109.06668/) (2021)
- [Generalization with Lossy Affordances: Leveraging Broad Offline Data for Learning Visuomotor Tasks](https://mlsanity.com/paper/2210.06601/) (2022)
- [MURAL: Meta-Learning Uncertainty-Aware Rewards for Outcome-Driven Reinforcement Learning](https://mlsanity.com/paper/2107.07184/) (2021)
- [Scalable Multi-Task Imitation Learning with Autonomous Improvement](https://mlsanity.com/paper/2003.02636/) (2020)
- [Hypernetworks for Zero-shot Transfer in Reinforcement Learning](https://mlsanity.com/paper/2211.15457/) (2022)
- [Would I have gotten that reward? Long-term credit assignment by counterfactual contribution analysis](https://mlsanity.com/paper/2306.16803/) (2023)
- [Episodic Self-Imitation Learning with Hindsight](https://mlsanity.com/paper/2011.13467/) (2020)
- [Guided Data Augmentation for Offline Reinforcement Learning and Imitation Learning](https://mlsanity.com/paper/2310.18247/) (2023)
- [DisCo RL: Distribution-Conditioned Reinforcement Learning for General-Purpose Policies](https://mlsanity.com/paper/2104.11707/) (2021)
- [Goal-Conditioned Generators of Deep Policies](https://mlsanity.com/paper/2207.01570/) (2022)
- [Hierarchical Reinforcement Learning for Air Combat at DARPA's AlphaDogfight Trials](https://mlsanity.com/paper/2105.00990/) (2021)
- [Hindsight Task Relabelling: Experience Replay for Sparse Reward Meta-RL](https://mlsanity.com/paper/2112.00901/) (2021)
- [Unbiased Methods for Multi-Goal Reinforcement Learning](https://mlsanity.com/paper/2106.08863/) (2021)
- [Diversity-based Trajectory and Goal Selection with Hindsight Experience Replay](https://mlsanity.com/paper/2108.07887/) (2021)
- [Hindsight Goal Ranking on Replay Buffer for Sparse Reward Environment](https://mlsanity.com/paper/2110.15043/) (2021)
- [Hidden Agenda: a Social Deduction Game with Diverse Learned Equilibria](https://mlsanity.com/paper/2201.01816/) (2022)
- [Skill Decision Transformer](https://mlsanity.com/paper/2301.13573/) (2023)
- [Understanding when Dynamics-Invariant Data Augmentations Benefit Model-Free Reinforcement Learning Updates](https://mlsanity.com/paper/2310.17786/) (2023)
- [Directed Exploration in Reinforcement Learning from Linear Temporal Logic](https://mlsanity.com/paper/2408.09495/) (2024)
- [Open-Ended Reinforcement Learning with Neural Reward Functions](https://mlsanity.com/paper/2202.08266/) (2022)
- [Offline Goal-Conditioned Reinforcement Learning for Safety-Critical Tasks with Recovery Policy](https://mlsanity.com/paper/2403.01734/) (2024)
- [BaNEL: Exploration Posteriors for Generative Modeling Using Only Negative Rewards](https://mlsanity.com/paper/2510.09596/) (2025)
- [Hindsight States: Blending Sim and Real Task Elements for Efficient Reinforcement Learning](https://mlsanity.com/paper/2303.02234/) (2023)
- [Hindsight Experience Replay Accelerates Proximal Policy Optimization](https://mlsanity.com/paper/2410.22524/) (2024)
- [RLFTSim: Realistic and Controllable Multi-Agent Traffic Simulation via Reinforcement Learning Fine-Tuning](https://mlsanity.com/paper/2605.19033/) (2026)
- [Goal-Conditioned Supervised Learning with Sub-Goal Prediction](https://mlsanity.com/paper/2305.10171/) (2023)

## Abstract (arXiv)

A reinforcement learning agent that needs to pursue different goals across episodes requires a goal-conditional policy. In addition to their potential to generalize desirable behavior to unseen goals, such policies may also enable higher-level planning based on subgoals. In sparse-reward environments, the capacity to exploit information about the degree to which an arbitrary goal has been achieved while another goal was intended appears crucial to enable sample efficient learning. However, reinforcement learning agents have only recently been endowed with such capacity for hindsight. In this paper, we demonstrate how hindsight can be introduced to policy gradient methods, generalizing this idea to a broad class of successful algorithms. Our experiments on a diverse selection of sparse-reward environments show that hindsight leads to a remarkable increase in sample efficiency.