# PlanGAN: Model-based Planning With Sparse Rewards and Multiple Goals

arXiv: https://arxiv.org/abs/2006.00900 · HTML: https://mlsanity.com/paper/2006.00900/

## Key facts

- Cited by 21 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 1 Jun 2020.

## Built on by (most cited first, 21 of 21)

- [Goal-Conditioned Reinforcement Learning: Problems and Solutions](https://mlsanity.com/paper/2201.08299/) (2022)
- [Planning to Practice: Efficient Online Fine-Tuning by Composing Goals in Latent Space](https://mlsanity.com/paper/2205.08129/) (2022)
- [GOPlan: Goal-conditioned Offline Reinforcement Learning by Planning with Learned Models](https://mlsanity.com/paper/2310.20025/) (2023)
- [MHER: Model-based Hindsight Experience Replay](https://mlsanity.com/paper/2107.00306/) (2021)
- [MapGo: Model-Assisted Policy Optimization for Goal-Oriented Tasks](https://mlsanity.com/paper/2105.06350/) (2021)
- [Learning to Shape Rewards using a Game of Two Partners](https://mlsanity.com/paper/2103.09159/) (2021)
- [Imaginary Hindsight Experience Replay: Curious Model-based Learning for Sparse Reward Tasks](https://mlsanity.com/paper/2110.02414/) (2021)
- [Looking Backward: Retrospective Backward Synthesis for Goal-Conditioned GFlowNets](https://mlsanity.com/paper/2406.01150/) (2024)
- [ReRoGCRL: Representation-based Robustness in Goal-Conditioned Reinforcement Learning](https://mlsanity.com/paper/2312.07392/) (2023)
- [Learning World Models for Unconstrained Goal Navigation](https://mlsanity.com/paper/2411.02446/) (2024)
- [Goal-conditioned Offline Reinforcement Learning through State Space Partitioning](https://mlsanity.com/paper/2303.09367/) (2023)
- [Goal-conditioned Offline Planning from Curious Exploration](https://mlsanity.com/paper/2311.16996/) (2023)
- [SVL: Goal-Conditioned Reinforcement Learning as Survival Learning](https://mlsanity.com/paper/2604.17551/) (2026)
- [FlowRL: Flow-Augmented Few-Shot Reinforcement Learning for Semi-Structured Sensor Data](https://mlsanity.com/paper/2409.14178/) (2024)
- [Generalized Bayesian deep reinforcement learning](https://mlsanity.com/paper/2412.11743/) (2024)
- [Goal-Conditioned Supervised Learning with Sub-Goal Prediction](https://mlsanity.com/paper/2305.10171/) (2023)
- [MRHER: Model-based Relay Hindsight Experience Replay for Sequential Object Manipulation Tasks with Sparse Rewards](https://mlsanity.com/paper/2306.16061/) (2023)
- [Data-Efficient Hierarchical Goal-Conditioned Reinforcement Learning via Normalizing Flows](https://mlsanity.com/paper/2602.11142/) (2026)
- [Automatic Evaluation of Excavator Operators using Learned Reward Functions](https://mlsanity.com/paper/2211.07941/) (2022)
- [ZeroDVFS: Zero-Shot LLM-Guided Core and Frequency Allocation for Embedded Platforms](https://mlsanity.com/paper/2601.08166/) (2026)
- [Improving Offline Goal-Conditioned Reinforcement Learning via Selective Reward Stimulation](https://mlsanity.com/paper/2609.19414/) (2026)

## Abstract (arXiv)

Learning with sparse rewards remains a significant challenge in reinforcement learning (RL), especially when the aim is to train a policy capable of achieving multiple different goals. To date, the most successful approaches for dealing with multi-goal, sparse reward environments have been model-free RL algorithms. In this work we propose PlanGAN, a model-based algorithm specifically designed for solving multi-goal tasks in environments with sparse rewards. Our method builds on the fact that any trajectory of experience collected by an agent contains useful information about how to achieve the goals observed during that trajectory. We use this to train an ensemble of conditional generative models (GANs) to generate plausible trajectories that lead the agent from its current state towards a specified goal. We then combine these imagined trajectories into a novel planning algorithm in order to achieve the desired goal as efficiently as possible. The performance of PlanGAN has been tested on a number of robotic navigation/manipulation tasks in comparison with a range of model-free reinforcement learning baselines, including Hindsight Experience Replay. Our studies indicate that PlanGAN can achieve comparable performance whilst being around 4-8 times more sample efficient.