# MapGo: Model-Assisted Policy Optimization for Goal-Oriented Tasks

arXiv: https://arxiv.org/abs/2105.06350 · HTML: https://mlsanity.com/paper/2105.06350/

## Key facts

- Cited by 13 papers in the map (data through 14 Oct 2026).
- Builds on 20 of its 30 references that are in the map.
- First posted on arXiv 13 May 2021.

## Builds on (most cited first, 20 of 20)

- [Model-Agnostic Meta-Learning for Fast Adaptation of Deep Networks](https://mlsanity.com/paper/1703.03400/) (2017)
- [Continuous control with deep reinforcement learning](https://mlsanity.com/paper/1509.02971/) (2015)
- [Hindsight Experience Replay](https://mlsanity.com/paper/1707.01495/) (2017)
- [Deep Reinforcement Learning in a Handful of Trials using Probabilistic Dynamics Models](https://mlsanity.com/paper/1805.12114/) (2018)
- [When to Trust Your Model: Model-Based Policy Optimization](https://mlsanity.com/paper/1906.08253/) (2019)
- [Automatic Goal Generation for Reinforcement Learning Agents](https://mlsanity.com/paper/1705.06366/) (2017)
- [Visual Reinforcement Learning with Imagined Goals](https://mlsanity.com/paper/1807.04742/) (2018)
- [Sample-Efficient Reinforcement Learning with Stochastic Ensemble Value Expansion](https://mlsanity.com/paper/1807.01675/) (2018)
- [Benchmarking Model-Based Reinforcement Learning](https://mlsanity.com/paper/1907.02057/) (2019)
- [Model-Based Value Estimation for Efficient Model-Free Reinforcement Learning](https://mlsanity.com/paper/1803.00101/) (2018)
- [Algorithmic Framework for Model-based Deep Reinforcement Learning with Theoretical Guarantees](https://mlsanity.com/paper/1807.03858/) (2018)
- [Maximum Entropy Gain Exploration for Long Horizon Multi-goal Reinforcement Learning](https://mlsanity.com/paper/2007.02832/) (2020)
- [Counterfactual Data Augmentation using Locally Factored Dynamics](https://mlsanity.com/paper/2007.02863/) (2020)
- [Exploring Model-based Planning with Policy Networks](https://mlsanity.com/paper/1906.08649/) (2019)
- [Keeping Your Distance: Solving Sparse Reward Tasks Using Self-Balancing Shaped Rewards](https://mlsanity.com/paper/1911.01417/) (2019)
- [Dynamical Distance Learning for Semi-Supervised and Unsupervised Skill Discovery](https://mlsanity.com/paper/1907.08225/) (2019)
- [Exploration via Hindsight Goal Generation](https://mlsanity.com/paper/1906.04279/) (2019)
- [Goal-Aware Prediction: Learning to Model What Matters](https://mlsanity.com/paper/2007.07170/) (2020)
- [PlanGAN: Model-based Planning With Sparse Rewards and Multiple Goals](https://mlsanity.com/paper/2006.00900/) (2020)
- [Regularizing Model-Based Planning with Energy-Based Models](https://mlsanity.com/paper/1910.05527/) (2019)

## Built on by (most cited first, 13 of 13)

- [Goal-Conditioned Reinforcement Learning: Problems and Solutions](https://mlsanity.com/paper/2201.08299/) (2022)
- [Diffusion Models for Reinforcement Learning: A Survey](https://mlsanity.com/paper/2311.01223/) (2023)
- [A Survey on Model-based Reinforcement Learning](https://mlsanity.com/paper/2206.09328/) (2022)
- [MHER: Model-based Hindsight Experience Replay](https://mlsanity.com/paper/2107.00306/) (2021)
- [Quality-Diversity Actor-Critic: Learning High-Performing and Diverse Behaviors via Value and Successor Features Critics](https://mlsanity.com/paper/2403.09930/) (2024)
- [Learning Diverse Policies in MOBA Games via Macro-Goals](https://mlsanity.com/paper/2110.14221/) (2021)
- [SVL: Goal-Conditioned Reinforcement Learning as Survival Learning](https://mlsanity.com/paper/2604.17551/) (2026)
- [Guided Cooperation in Hierarchical Reinforcement Learning via Model-based Rollout](https://mlsanity.com/paper/2309.13508/) (2023)
- [MRHER: Model-based Relay Hindsight Experience Replay for Sequential Object Manipulation Tasks with Sparse Rewards](https://mlsanity.com/paper/2306.16061/) (2023)
- [Data-Efficient Hierarchical Goal-Conditioned Reinforcement Learning via Normalizing Flows](https://mlsanity.com/paper/2602.11142/) (2026)
- [Planning Immediate Landmarks of Targets for Model-Free Skill Transfer across Agents](https://mlsanity.com/paper/2212.09033/) (2022)
- [Goal-Conditioned Terminal Value Estimation for Real-time and Multi-task Model Predictive Control](https://mlsanity.com/paper/2410.04929/) (2024)
- [Multi-Goal Dexterous Hand Manipulation using Probabilistic Model-based Reinforcement Learning](https://mlsanity.com/paper/2504.21585/) (2025)

## Abstract (arXiv)

In Goal-oriented Reinforcement learning, relabeling the raw goals in past experience to provide agents with hindsight ability is a major solution to the reward sparsity problem. In this paper, to enhance the diversity of relabeled goals, we develop FGI (Foresight Goal Inference), a new relabeling strategy that relabels the goals by looking into the future with a learned dynamics model. Besides, to improve sample efficiency, we propose to use the dynamics model to generate simulated trajectories for policy training. By integrating these two improvements, we introduce the MapGo framework (Model-Assisted Policy Optimization for Goal-oriented tasks). In our experiments, we first show the effectiveness of the FGI strategy compared with the hindsight one, and then show that the MapGo framework achieves higher sample efficiency when compared to model-free baselines on a set of complicated tasks.