mlsanityData to 7 Oct 2026

Rewriting History with Inverse RL: Hindsight Inference for Policy Improvement

arXiv 2002.11089 ↗First posted 25 Feb 2020cs.LG cs.AI cs.ROcited by 59

Not on the map. Why?

Cited by 28 in Offline RL and imitation learning from data, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

Multi-task reinforcement learning (RL) aims to simultaneously learn policies for solving many tasks. Several prior works have found that relabeling past experience with different reward functions can improve sample efficiency. Relabeling methods typically ask: if, in hindsight, we assume that our experience was optimal for some task, for what task was it optimal? In this paper, we show that hindsight relabeling is inverse RL, an observation that suggests that we can use inverse RL in tandem for RL algorithms to efficiently solve many tasks. We use this idea to generalize goal-relabeling techniques from prior work to arbitrary classes of tasks. Our experiments confirm that relabeling data using inverse RL accelerates learning in general multi-task settings, including goal-reaching, domains with discrete sets of rewards, and those with linear reward functions.

Key facts. Cited by 59 papers in the map (data through 7 Oct 2026). First posted on arXiv 25 Feb 2020.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 15; 2022: 25; 2023: 8; 2024: 5; 2025: 2; 2026: 4.

59 papers.

Sort:

By area:

  1. 2022Contrastive Learning as Goal-Conditioned Reinforcement Learningcited by 241
  2. 2021MT-Opt: Continuous Multi-Task Robotic Reinforcement Learning at Scalecited by 226
  3. 2021RvS: What is Essential for Offline RL via Supervised Learning?cited by 184
  4. 2022Goal-Conditioned Reinforcement Learning: Problems and Solutionscited by 145
  5. 2021Actionable Models: Unsupervised Offline Reinforcement Learning of Robotic Skillscited by 125
  6. 2021Generalized Decision Transformer for Offline Hindsight Information Matchingcited by 105
  7. 2022Don't Change the Algorithm, Change the Data: Exploratory Data for Offline Reinforcement Learningcited by 93
  8. 2022Rethinking Goal-conditioned Supervised Learning and Its Connection to Offline RLcited by 89
  9. 2022How Far I'll Go: Offline Goal-Conditioned Reinforcement Learning via f-Advantage Regressioncited by 65
  10. 2021Conservative Data Sharing for Multi-Task Offline Reinforcement Learningcited by 64
  11. 2022How to Leverage Unlabeled Data in Offline Reinforcement Learningcited by 61
  12. 2022Bisimulation Makes Analogies in Goal-Conditioned Reinforcement Learningcited by 38
38 more
  1. 2023What is Essential for Unseen Goal Generalization of Offline Goal-conditioned RL?cited by 36
  2. 2023AMAGO: Scalable In-Context Reinforcement Learning for Adaptive Agentscited by 36
  3. 2022On Transforming Reinforcement Learning by Transformer: The Development Trajectorycited by 35
  4. 2022Planning to Practice: Efficient Online Fine-Tuning by Composing Goals in Latent Spacecited by 31
  5. 2025Beyond Verifiable Rewards: Scaling Reinforcement Learning for Language Models to Unverifiable Datacited by 29
  6. 2022Offline Reinforcement Learning for Visual Navigationcited by 28
  7. 2021StARformer: Transformer with State-Action-Reward Representations for Visual Reinforcement Learningcited by 27
  8. 2022The Challenges of Exploration for Offline Reinforcement Learningcited by 26
  9. 2022Generalization with Lossy Affordances: Leveraging Broad Offline Data for Learning Visuomotor Taskscited by 24
  10. 2022Context-Aware Language Modeling for Goal-Oriented Dialogue Systemscited by 22
  11. 2023Waypoint Transformer: Reinforcement Learning via Supervised Learning with Intermediate Targetscited by 22
  12. 2022Imitating Past Successes can be Very Suboptimalcited by 20
  13. 2023Graph Decision Transformercited by 20
  14. 2023Distance Weighted Supervised Learning for Offline Interaction Datacited by 18
  15. 2021Computational Benefits of Intermediate Rewards for Goal-Reaching Policy Learningcited by 17
  16. 2023Interpretable Reward Redistribution in Reinforcement Learning: A Causal Approachcited by 17
  17. 2022Understanding the Complexity Gains of Single-Task RL with a Curriculumcited by 16
  18. 2021MHER: Model-based Hindsight Experience Replaycited by 14
  19. 2024Offline Multitask Representation Learning for Reinforcement Learningcited by 14
  20. 2021Wish you were here: Hindsight Goal Selection for long-horizon dexterous manipulationcited by 13
  21. 2023Hundreds Guide Millions: Adaptive Offline Reinforcement Learning with Expert Guidancecited by 13
  22. 2022A System for Morphology-Task Generalization via Unified Representation and Behavior Distillationcited by 12
  23. 2022Switch Trajectory Transformer with Distributional Value Approximation for Multi-Task Reinforcement Learningcited by 11
  24. 2022Automating Reinforcement Learning with Example-based Resetscited by 11
  25. 2022One After Another: Learning Incremental Skills for a Changing Worldcited by 10
  26. 2021DisCo RL: Distribution-Conditioned Reinforcement Learning for General-Purpose Policiescited by 9
  27. 2021Hindsight Task Relabelling: Experience Replay for Sparse Reward Meta-RLcited by 8
  28. 2023Design from Policies: Conservative Test-Time Adaptation for Offline Policy Optimizationcited by 8
  29. 2021Hindsight Goal Ranking on Replay Buffer for Sparse Reward Environmentcited by 7
  30. 2024Pessimistic Value Iteration for Multi-Task Data Sharing in Offline Reinforcement Learningcited by 6
  31. 2022Stein Variational Goal Generation for adaptive Exploration in Multi-Goal Reinforcement Learningcited by 5
  32. 2022Learning Multi-Task Transferable Rewards via Variational Inverse Reinforcement Learningcited by 5
  33. 2021Hindsight Foresight Relabeling for Meta-Reinforcement Learningcited by 4
  34. 2021Learning from demonstrations with SACR2: Soft Actor-Critic with Reward Relabelingcited by 4
  35. 2021Provably Improved Context-Based Offline Meta-RL with Attention and Contrastive Learningcited by 2
  36. 2024Urban-Focused Multi-Task Offline Reinforcement Learning with Contrastive Data Sharingcited by 2
  37. 2024Leveraging Unlabeled Data Sharing through Kernel Function Approximation in Offline Reinforcement Learningcited by 2
  38. 2022STIR2: Reward Relabelling for combined Reinforcement and Imitation Learning on sparse-reward taskscited by 1

50 most cited of 59 All 59 (JSON)