Hindsight Expectation Maximization for Goal-conditioned Reinforcement Learning
Not on the map. Why?
Abstract (arXiv)
We propose a graphical model framework for goal-conditioned RL, with an EM algorithm that operates on the lower bound of the RL objective. The E-step provides a natural interpretation of how 'learning in hindsight' techniques, such as HER, to handle extremely sparse goal-conditioned rewards. The M-step reduces policy optimization to supervised learning updates, which greatly stabilizes end-to-end training on high-dimensional inputs such as images. We show that the combined algorithm, hEM significantly outperforms model-free baselines on a wide range of goal-conditioned benchmarks with sparse rewards.
Key facts. Cited by 12 papers in the map (data through 7 Oct 2026). First posted on arXiv 13 Jun 2020.
Builds on
No reference list yet.
Built on by
Citing papers per year: 2021: 2; 2022: 5; 2023: 1; 2024: 2; 2025: 2.
12 papers.
- 2022Goal-Conditioned Reinforcement Learning: Problems and Solutionscited by 145
- 2022Generalizing Goal-Conditioned Reinforcement Learning with Variational Causal Reasoningcited by 33
- 2025Beyond Verifiable Rewards: Scaling Reinforcement Learning for Language Models to Unverifiable Datacited by 29
- 2021Co-Adaptation of Algorithmic and Implementational Innovations in Inference-based Deep Reinforcement Learningcited by 15
- 2022Cell-Free Latent Go-Explorecited by 5
- 2023Visual Hindsight Self-Imitation Learning for Interactive Navigationcited by 5
- 2021Learning for Visual Navigation by Imagining the Successcited by 2
- 2022Deriving time-averaged active inference from control principlescited by 1
- 2024Attaining Human`s Desirable Outcomes in Human-AI Interaction via Structural Causal Gamescited by 1
- 2025Dynamic Object Masks as Goal Representations for Visual Goal-Conditioned Reinforcement Learningcited by 1
- 2022Understanding Hindsight Goal Relabeling from a Divergence Minimization Perspectivecited by 0
- 2024Goal-Conditioned Terminal Value Estimation for Real-time and Multi-task Model Predictive Controlcited by 0
- 2025Dynamic Object Masks as Goal Representations for Visual Goal-Conditioned Reinforcement Learning6 Oct 2025
- 2025Beyond Verifiable Rewards: Scaling Reinforcement Learning for Language Models to Unverifiable Data25 Mar 2025
- 2024Goal-Conditioned Terminal Value Estimation for Real-time and Multi-task Model Predictive Control7 Oct 2024
- 2024Attaining Human`s Desirable Outcomes in Human-AI Interaction via Structural Causal Games26 May 2024
- 2023Visual Hindsight Self-Imitation Learning for Interactive Navigation5 Dec 2023
- 2022Understanding Hindsight Goal Relabeling from a Divergence Minimization Perspective26 Sep 2022
- 2022Cell-Free Latent Go-Explore31 Aug 2022
- 2022Deriving time-averaged active inference from control principles22 Aug 2022
- 2022Generalizing Goal-Conditioned Reinforcement Learning with Variational Causal Reasoning19 Jul 2022
- 2022Goal-Conditioned Reinforcement Learning: Problems and Solutions20 Jan 2022
- 2021Co-Adaptation of Algorithmic and Implementational Innovations in Inference-based Deep Reinforcement Learning31 Mar 2021
- 2021Learning for Visual Navigation by Imagining the Success28 Feb 2021