mlsanityData to 7 Oct 2026

Understanding Hindsight Goal Relabeling from a Divergence Minimization Perspective

arXiv 2209.13046 ↗First posted 26 Sep 2022cs.LG cs.AIcited by 0

On the map: Offline RL and imitation learning from data → Imitation learning methods

Abstract (arXiv)

Hindsight goal relabeling has become a foundational technique in multi-goal reinforcement learning (RL). The essential idea is that any trajectory can be seen as a sub-optimal demonstration for reaching its final state. Intuitively, learning from those arbitrary demonstrations can be seen as a form of imitation learning (IL). However, the connection between hindsight goal relabeling and imitation learning is not well understood. In this paper, we propose a novel framework to understand hindsight goal relabeling from a divergence minimization perspective. Recasting the goal reaching problem in the IL framework not only allows us to derive several existing methods from first principles, but also provides us with the tools from IL to improve goal reaching algorithms. Experimentally, we find that under hindsight relabeling, Q-learning outperforms behavioral cloning (BC). Yet, a vanilla combination of both hurts performance. Concretely, we see that the BC loss only helps when selectively applied to actions that get the agent closer to the goal according to the Q-function. Our framework also explains the puzzling phenomenon wherein a reward of (-1, 0) results in significantly better performance than a (0, 1) reward for goal reaching.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Placed in Imitation learning methods, one of 13 directions in Offline RL and imitation learning from data. Builds on 56 of its 74 references that are in the map. First posted on arXiv 26 Sep 2022.

Builds on

56 of 74 references.

Closest ancestors · 10

Same direction, newest first

  1. 2021IQ-Learn: Inverse soft-Q Learning for Imitation23 Jun
  2. 2021Adversarial Intrinsic Motivation for Reinforcement Learning27 May
  3. 2021Off-Policy Imitation Learning from Observations25 Feb
  4. 2020f-GAIL: Learning f-Divergence for Generative Adversarial Imitation Learning2 Oct
  5. 2019Learning to Reach Goals via Iterated Supervised Learning12 Dec
  6. 2019Imitation Learning via Off-Policy Distribution Matching10 Dec
  7. 2019A Divergence Minimization Perspective on Imitation Learning Methods6 Nov
  8. 2019Goal-conditioned Imitation Learning13 Jun
  9. 2019Imitation Learning as f-Divergence Minimization30 May
  10. 2018Self-Imitation Learning14 Jun

Other references · 46

  1. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  2. 2020Language Models are Few-Shot Learnerscited by 32,199
  3. 2017Proximal Policy Optimization Algorithmscited by 14,441
  4. 2018Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actorcited by 4,635
  5. 2015Continuous control with deep reinforcement learningcited by 3,279
  6. 2018Addressing Function Approximation Error in Actor-Critic Methodscited by 2,302
  7. 2022Do As I Can, Not As I Say: Grounding Language in Robotic Affordancescited by 2,302
  8. 2015Deep Reinforcement Learning with Double Q-learningcited by 1,704
  9. 2016Generative Adversarial Imitation Learningcited by 1,576
  10. 2017Hindsight Experience Replaycited by 1,096
  11. 2022BC-Z: Zero-Shot Task Generalization with Robotic Imitation Learningcited by 517
  12. 2016f-GAN: Training Generative Neural Samplers using Variational Divergence Minimizationcited by 504
34 more
  1. 2017Learning Robust Rewards with Adversarial Inverse Reinforcement Learningcited by 485
  2. 2018Reinforcement Learning and Control as Probabilistic Inference: Tutorial and Reviewcited by 466
  3. 2017Overcoming Exploration in Reinforcement Learning with Demonstrationscited by 322
  4. 2019Learning Latent Plans from Playcited by 310
  5. 2022Behavior Transformers: Cloning k modes with one stonecited by 299
  6. 2018Multi-Goal Reinforcement Learning: Challenging Robotics Environments and Request for Researchcited by 256
  7. 2019DualDICE: Behavior-Agnostic Estimation of Discounted Stationary Distribution Correctionscited by 254
  8. 2018Visual Reinforcement Learning with Imagined Goalscited by 248
  9. 2022Contrastive Learning as Goal-Conditioned Reinforcement Learningcited by 241
  10. 2017One-Shot Imitation Learningcited by 232
  11. 2020Language Conditioned Imitation Learning over Unstructured Datacited by 208
  12. 2019AlgaeDICE: Policy Gradient from Arbitrary Experiencecited by 187
  13. 2017Reverse Curriculum Generation for Reinforcement Learningcited by 185
  14. 2017Equivalence Between Policy Gradients and Soft Q-Learningcited by 172
  15. 2019Skew-Fit: State-Covering Self-Supervised Reinforcement Learningcited by 171
  16. 2016A Connection between Generative Adversarial Networks, Inverse Reinforcement Learning, and Energy-Based Modelscited by 167
  17. 2020Rearrangement: A Challenge for Embodied AIcited by 153
  18. 2021Actionable Models: Unsupervised Offline Reinforcement Learning of Robotic Skillscited by 125
  19. 2018Unsupervised Control Through Non-Parametric Discriminative Rewardscited by 102
  20. 2020Reinforcement Learning via Fenchel-Rockafellar Dualitycited by 100
  21. 2020Maximum Entropy Gain Exploration for Long Horizon Multi-goal Reinforcement Learningcited by 100
  22. 2020C-Learning: Learning to Achieve Goals via Recursive Classificationcited by 99
  23. 2018Temporal Difference Models: Model-Free Deep RL for Model-Based Controlcited by 89
  24. 2021Asymmetric self-play for automatic goal discovery in robotic manipulationcited by 71
  25. 2019ROBEL: Robotics Benchmarks for Learning with Low-Cost Robotscited by 70
  26. 2020World Model as a Graph: Learning Latent Landmarks for Planningcited by 67
  27. 2021Learning Successor States and Goal-Dependent Values: A Mathematical Viewpointcited by 67
  28. 2022How Far I'll Go: Offline Goal-Conditioned Reinforcement Learning via f-Advantage Regressioncited by 65
  29. 2020Rewriting History with Inverse RL: Hindsight Inference for Policy Improvementcited by 59
  30. 2016Model-Free Imitation Learning with Policy Optimizationcited by 52
  31. 2021Outcome-Driven Reinforcement Learning via Variational Inferencecited by 17
  32. 2019Reinforcement Learning without Ground-Truth Statecited by 15
  33. 2020Hindsight Expectation Maximization for Goal-conditioned Reinforcement Learningcited by 12
  34. 2022Bilinear value networkscited by 9

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2020Generalized Hindsight for Reinforcement Learning
  2. 2026An Informational Curse of Horizon in Goal-Conditioned Policy Learning
  3. 2026Where the Model Changes Its Mind: Hindsight-Divergence Localization for Efficient Reinforcement Learning with Verifiable Rewards
  4. 2026Learning More from Less: Reinforcement Learning from Hindsight
  5. 2023Visual Hindsight Self-Imitation Learning for Interactive Navigation
  6. 2023Backward Learning for Goal-Conditioned Policies
  7. 2025Autonomous Learning From Success and Failure: Goal-Conditioned Supervised Learning with Negative Feedback
  8. 2019Learning from Trajectories via Subgoal Discovery