# Maximum Entropy-Regularized Multi-Goal Reinforcement Learning

arXiv: https://arxiv.org/abs/1905.08786 · HTML: https://mlsanity.com/paper/1905.08786/

## Key facts

- Cited by 40 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 21 May 2019.

## Built on by (most cited first, 30 of 40)

- [Contrastive Learning as Goal-Conditioned Reinforcement Learning](https://mlsanity.com/paper/2206.07568/) (2022)
- [Goal-Conditioned Reinforcement Learning with Imagined Subgoals](https://mlsanity.com/paper/2107.00541/) (2021)
- [From Play to Policy: Conditional Behavior Generation from Uncurated Robot Data](https://mlsanity.com/paper/2210.10047/) (2022)
- [BYOL-Explore: Exploration by Bootstrapped Prediction](https://mlsanity.com/paper/2206.08332/) (2022)
- [Maximum Entropy Population-Based Training for Zero-Shot Human-AI Coordination](https://mlsanity.com/paper/2112.11701/) (2021)
- [Exploration in Deep Reinforcement Learning: From Single-Agent to Multiagent Domain](https://mlsanity.com/paper/2109.06668/) (2021)
- [What is Essential for Unseen Goal Generalization of Offline Goal-conditioned RL?](https://mlsanity.com/paper/2305.18882/) (2023)
- [An information-theoretic perspective on intrinsic motivation in reinforcement learning: a survey](https://mlsanity.com/paper/2209.08890/) (2022)
- [Efficient Continuous Control with Double Actors and Regularized Critics](https://mlsanity.com/paper/2106.03050/) (2021)
- [Mutual Information State Intrinsic Control](https://mlsanity.com/paper/2103.08107/) (2021)
- [Phasic Self-Imitative Reduction for Sparse-Reward Goal-Conditioned Reinforcement Learning](https://mlsanity.com/paper/2206.12030/) (2022)
- [ACE : Off-Policy Actor-Critic with Causality-Aware Entropy Regularization](https://mlsanity.com/paper/2402.14528/) (2024)
- [MHER: Model-based Hindsight Experience Replay](https://mlsanity.com/paper/2107.00306/) (2021)
- [EntroPIC: Towards Stable Long-Term Training of LLMs via Entropy Stabilization with Proportional-Integral Control](https://mlsanity.com/paper/2511.15248/) (2025)
- [HYPER: Learned Hybrid Trajectory Prediction via Factored Inference and Adaptive Sampling](https://mlsanity.com/paper/2110.02344/) (2021)
- [Beyond Supervised Continual Learning: a Review](https://mlsanity.com/paper/2208.14307/) (2022)
- [Learn the Ropes, Then Trust the Wins: Self-imitation with Progressive Exploration for Agentic Reinforcement Learning](https://mlsanity.com/paper/2509.22601/) (2025)
- [DisTop: Discovering a Topological representation to learn diverse and rewarding skills](https://mlsanity.com/paper/2106.03853/) (2021)
- [Goal-Aware Cross-Entropy for Multi-Target Reinforcement Learning](https://mlsanity.com/paper/2110.12985/) (2021)
- [Hindsight Goal Ranking on Replay Buffer for Sparse Reward Environment](https://mlsanity.com/paper/2110.15043/) (2021)
- [Null Counterfactual Factor Interactions for Goal-Conditioned Reinforcement Learning](https://mlsanity.com/paper/2505.03172/) (2025)
- [Performative Policy Gradient: Optimality in Performative Reinforcement Learning](https://mlsanity.com/paper/2512.20576/) (2025)
- [Bias-reduced Multi-step Hindsight Experience Replay for Efficient Multi-goal Reinforcement Learning](https://mlsanity.com/paper/2102.12962/) (2021)
- [Adaptive Multi-Goal Exploration](https://mlsanity.com/paper/2111.12045/) (2021)
- [Evolution Guided Generative Flow Networks](https://mlsanity.com/paper/2402.02186/) (2024)
- [Improving Environment Novelty Quantification for Effective Unsupervised Environment Design](https://mlsanity.com/paper/2502.05726/) (2025)
- [LLM-Explorer: A Plug-in Reinforcement Learning Policy Exploration Enhancement Driven by Large Language Models](https://mlsanity.com/paper/2505.15293/) (2025)
- [L-SA: Learning Under-Explored Targets in Multi-Target Reinforcement Learning](https://mlsanity.com/paper/2305.13741/) (2023)
- [Contact Energy Based Hindsight Experience Prioritization](https://mlsanity.com/paper/2312.02677/) (2023)
- [Constrained Sliced Wasserstein Embedding](https://mlsanity.com/paper/2506.02203/) (2025)

## Abstract (arXiv)

In Multi-Goal Reinforcement Learning, an agent learns to achieve multiple goals with a goal-conditioned policy. During learning, the agent first collects the trajectories into a replay buffer, and later these trajectories are selected randomly for replay. However, the achieved goals in the replay buffer are often biased towards the behavior policies. From a Bayesian perspective, when there is no prior knowledge about the target goal distribution, the agent should learn uniformly from diverse achieved goals. Therefore, we first propose a novel multi-goal RL objective based on weighted entropy. This objective encourages the agent to maximize the expected return, as well as to achieve more diverse goals. Secondly, we developed a maximum entropy-based prioritization framework to optimize the proposed objective. For evaluation of this framework, we combine it with Deep Deterministic Policy Gradient, both with or without Hindsight Experience Replay. On a set of multi-goal robotic tasks of OpenAI Gym, we compare our method with other baselines and show promising improvements in both performance and sample-efficiency.