# Hindsight Expectation Maximization for Goal-conditioned Reinforcement Learning

arXiv: https://arxiv.org/abs/2006.07549 · HTML: https://mlsanity.com/paper/2006.07549/

## Key facts

- Cited by 12 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 13 Jun 2020.

## Built on by (most cited first, 12 of 12)

- [Goal-Conditioned Reinforcement Learning: Problems and Solutions](https://mlsanity.com/paper/2201.08299/) (2022)
- [Generalizing Goal-Conditioned Reinforcement Learning with Variational Causal Reasoning](https://mlsanity.com/paper/2207.09081/) (2022)
- [Beyond Verifiable Rewards: Scaling Reinforcement Learning for Language Models to Unverifiable Data](https://mlsanity.com/paper/2503.19618/) (2025)
- [Co-Adaptation of Algorithmic and Implementational Innovations in Inference-based Deep Reinforcement Learning](https://mlsanity.com/paper/2103.17258/) (2021)
- [Visual Hindsight Self-Imitation Learning for Interactive Navigation](https://mlsanity.com/paper/2312.03446/) (2023)
- [Cell-Free Latent Go-Explore](https://mlsanity.com/paper/2208.14928/) (2022)
- [Learning for Visual Navigation by Imagining the Success](https://mlsanity.com/paper/2103.00446/) (2021)
- [Deriving time-averaged active inference from control principles](https://mlsanity.com/paper/2208.10601/) (2022)
- [Attaining Human`s Desirable Outcomes in Human-AI Interaction via Structural Causal Games](https://mlsanity.com/paper/2405.16588/) (2024)
- [Dynamic Object Masks as Goal Representations for Visual Goal-Conditioned Reinforcement Learning](https://mlsanity.com/paper/2510.06277/) (2025)
- [Understanding Hindsight Goal Relabeling from a Divergence Minimization Perspective](https://mlsanity.com/paper/2209.13046/) (2022)
- [Goal-Conditioned Terminal Value Estimation for Real-time and Multi-task Model Predictive Control](https://mlsanity.com/paper/2410.04929/) (2024)

## Abstract (arXiv)

We propose a graphical model framework for goal-conditioned RL, with an EM algorithm that operates on the lower bound of the RL objective. The E-step provides a natural interpretation of how 'learning in hindsight' techniques, such as HER, to handle extremely sparse goal-conditioned rewards. The M-step reduces policy optimization to supervised learning updates, which greatly stabilizes end-to-end training on high-dimensional inputs such as images. We show that the combined algorithm, hEM significantly outperforms model-free baselines on a wide range of goal-conditioned benchmarks with sparse rewards.