# Generalized Hindsight for Reinforcement Learning

arXiv: https://arxiv.org/abs/2002.11708 · HTML: https://mlsanity.com/paper/2002.11708/

## Key facts

- Cited by 51 papers in the map (data through 7 Oct 2026).
- First posted on arXiv 26 Feb 2020.

## Built on by (most cited first, 30 of 51)

- [Contrastive Learning as Goal-Conditioned Reinforcement Learning](https://mlsanity.com/paper/2206.07568/) (2022)
- [Rewards-in-Context: Multi-objective Alignment of Foundation Models with Dynamic Preference Adjustment](https://mlsanity.com/paper/2402.10207/) (2024)
- [HIQL: Offline Goal-Conditioned RL with Latent States as Actions](https://mlsanity.com/paper/2307.11949/) (2023)
- [Generalized Decision Transformer for Offline Hindsight Information Matching](https://mlsanity.com/paper/2111.10364/) (2021)
- [Don't Change the Algorithm, Change the Data: Exploratory Data for Offline Reinforcement Learning](https://mlsanity.com/paper/2201.13425/) (2022)
- [Rethinking Goal-conditioned Supervised Learning and Its Connection to Offline RL](https://mlsanity.com/paper/2202.04478/) (2022)
- [Conservative Data Sharing for Multi-Task Offline Reinforcement Learning](https://mlsanity.com/paper/2109.08128/) (2021)
- [How to Leverage Unlabeled Data in Offline Reinforcement Learning](https://mlsanity.com/paper/2202.01741/) (2022)
- [What is Essential for Unseen Goal Generalization of Offline Goal-conditioned RL?](https://mlsanity.com/paper/2305.18882/) (2023)
- [On Transforming Reinforcement Learning by Transformer: The Development Trajectory](https://mlsanity.com/paper/2212.14164/) (2022)
- [Unsupervised Zero-Shot Reinforcement Learning via Functional Reward Encodings](https://mlsanity.com/paper/2402.17135/) (2024)
- [Cross-Domain Policy Adaptation via Value-Guided Data Filtering](https://mlsanity.com/paper/2305.17625/) (2023)
- [StARformer: Transformer with State-Action-Reward Representations for Visual Reinforcement Learning](https://mlsanity.com/paper/2110.06206/) (2021)
- [The Challenges of Exploration for Offline Reinforcement Learning](https://mlsanity.com/paper/2201.11861/) (2022)
- [Imitating Past Successes can be Very Suboptimal](https://mlsanity.com/paper/2206.03378/) (2022)
- [Graph Decision Transformer](https://mlsanity.com/paper/2303.03747/) (2023)
- [Distance Weighted Supervised Learning for Offline Interaction Data](https://mlsanity.com/paper/2304.13774/) (2023)
- [Reinforcing Language Agents via Policy Optimization with Action Decomposition](https://mlsanity.com/paper/2405.15821/) (2024)
- [MHER: Model-based Hindsight Experience Replay](https://mlsanity.com/paper/2107.00306/) (2021)
- [It Takes Four to Tango: Multiagent Selfplay for Automatic Curriculum Generation](https://mlsanity.com/paper/2202.10608/) (2022)
- [Offline Multitask Representation Learning for Reinforcement Learning](https://mlsanity.com/paper/2403.11574/) (2024)
- [Wish you were here: Hindsight Goal Selection for long-horizon dexterous manipulation](https://mlsanity.com/paper/2112.00597/) (2021)
- [A System for Morphology-Task Generalization via Unified Representation and Behavior Distillation](https://mlsanity.com/paper/2211.14296/) (2022)
- [Visual Affordance Prediction for Guiding Robot Exploration](https://mlsanity.com/paper/2305.17783/) (2023)
- [Entropy-Regularized Token-Level Policy Optimization for Language Agent Reinforcement](https://mlsanity.com/paper/2402.06700/) (2024)
- [One After Another: Learning Incremental Skills for a Changing World](https://mlsanity.com/paper/2203.11176/) (2022)
- [The Provable Benefits of Unsupervised Data Sharing for Offline Reinforcement Learning](https://mlsanity.com/paper/2302.13493/) (2023)
- [Hindsight Task Relabelling: Experience Replay for Sparse Reward Meta-RL](https://mlsanity.com/paper/2112.00901/) (2021)
- [Hindsight Goal Ranking on Replay Buffer for Sparse Reward Environment](https://mlsanity.com/paper/2110.15043/) (2021)
- [Understanding when Dynamics-Invariant Data Augmentations Benefit Model-Free Reinforcement Learning Updates](https://mlsanity.com/paper/2310.17786/) (2023)

## Abstract (arXiv)

One of the key reasons for the high sample complexity in reinforcement learning (RL) is the inability to transfer knowledge from one task to another. In standard multi-task RL settings, low-reward data collected while trying to solve one task provides little to no signal for solving that particular task and is hence effectively wasted. However, we argue that this data, which is uninformative for one task, is likely a rich source of information for other tasks. To leverage this insight and efficiently reuse data, we present Generalized Hindsight: an approximate inverse reinforcement learning technique for relabeling behaviors with the right tasks. Intuitively, given a behavior generated under one task, Generalized Hindsight returns a different task that the behavior is better suited for. Then, the behavior is relabeled with this new task before being used by an off-policy RL optimizer. Compared to standard relabeling techniques, Generalized Hindsight provides a substantially more efficient reuse of samples, which we empirically demonstrate on a suite of multi-task navigation and manipulation tasks. Videos and code can be accessed here: https://sites.google.com/view/generalized-hindsight.