# Goal-Conditioned Reinforcement Learning with Disentanglement-based Reachability Planning

arXiv: https://arxiv.org/abs/2307.10846 · HTML: https://mlsanity.com/paper/2307.10846/

## Key facts

- Cited by 5 papers in the map (data through 14 Oct 2026).
- Builds on 28 of its 35 references that are in the map.
- First posted on arXiv 20 Jul 2023.

## Builds on (most cited first, 28 of 28)

- [Addressing Function Approximation Error in Actor-Critic Methods](https://mlsanity.com/paper/1802.09477/) (2018)
- [Spatial Transformer Networks](https://mlsanity.com/paper/1506.02025/) (2015)
- [Curiosity-driven Exploration by Self-supervised Prediction](https://mlsanity.com/paper/1705.05363/) (2017)
- [Hindsight Experience Replay](https://mlsanity.com/paper/1707.01495/) (2017)
- [Exploration by Random Network Distillation](https://mlsanity.com/paper/1810.12894/) (2018)
- [CURL: Contrastive Unsupervised Representations for Reinforcement Learning](https://mlsanity.com/paper/2004.04136/) (2020)
- [robosuite: A Modular Simulation Framework and Benchmark for Robot Learning](https://mlsanity.com/paper/2009.12293/) (2020)
- [#Exploration: A Study of Count-Based Exploration for Deep Reinforcement Learning](https://mlsanity.com/paper/1611.04717/) (2016)
- [Visual Reinforcement Learning with Imagined Goals](https://mlsanity.com/paper/1807.04742/) (2018)
- [Search on the Replay Buffer: Bridging Planning and Reinforcement Learning](https://mlsanity.com/paper/1906.05253/) (2019)
- [SPACE: Unsupervised Object-Oriented Scene Representation via Spatial Attention and Decomposition](https://mlsanity.com/paper/2001.02407/) (2020)
- [Goal-Conditioned Reinforcement Learning: Problems and Solutions](https://mlsanity.com/paper/2201.08299/) (2022)
- [Planning with Goal-Conditioned Policies](https://mlsanity.com/paper/1911.08453/) (2019)
- [Goal-Conditioned Reinforcement Learning with Imagined Subgoals](https://mlsanity.com/paper/2107.00541/) (2021)
- [Episodic Curiosity through Reachability](https://mlsanity.com/paper/1810.02274/) (2018)
- [SCALOR: Generative World Models with Scalable Object Representations](https://mlsanity.com/paper/1910.02384/) (2019)
- [Temporal Difference Models: Model-Free Deep RL for Model-Based Control](https://mlsanity.com/paper/1802.09081/) (2018)
- [Autotelic Agents with Intrinsically Motivated Goal-Conditioned Reinforcement Learning: a Short Survey](https://mlsanity.com/paper/2012.09830/) (2020)
- [ViNG: Learning Open-World Navigation with Visual Goals](https://mlsanity.com/paper/2012.09812/) (2020)
- [World Model as a Graph: Learning Latent Landmarks for Planning](https://mlsanity.com/paper/2011.12491/) (2020)
- [SORNet: Spatial Object-Centric Representations for Sequential Manipulation](https://mlsanity.com/paper/2109.03891/) (2021)
- [IFOR: Iterative Flow Minimization for Robotic Object Rearrangement](https://mlsanity.com/paper/2202.00732/) (2022)
- [What Can I Do Here? Learning New Skills by Imagining Visual Affordances](https://mlsanity.com/paper/2106.00671/) (2021)
- [C-Planning: An Automatic Curriculum for Learning Goal-Reaching Tasks](https://mlsanity.com/paper/2110.12080/) (2021)
- [LECO: Learnable Episodic Count for Task-Specific Intrinsic Reward](https://mlsanity.com/paper/2210.05409/) (2022)
- [ROLL: Visual Self-Supervised Reinforcement Learning with Object Reasoning](https://mlsanity.com/paper/2011.06777/) (2020)
- [Hindsight Task Relabelling: Experience Replay for Sparse Reward Meta-RL](https://mlsanity.com/paper/2112.00901/) (2021)
- [Weakly Supervised Disentangled Representation for Goal-conditioned Reinforcement Learning](https://mlsanity.com/paper/2202.13624/) (2022)

## Built on by (most cited first, 5 of 5)

- [Predictive but Not Plannable: RC-aux for Latent World Models](https://mlsanity.com/paper/2605.07278/) (2026)
- [World Model Control by Trajectory Reachability Metrics](https://mlsanity.com/paper/2605.22164/) (2026)
- [Hierarchical Reinforcement Learning for Articulated Tool Manipulation with Multifingered Hand](https://mlsanity.com/paper/2507.06822/) (2025)
- [Action from Adjacent Set in Physical Space Outperforms the Best Prediction in World Models](https://mlsanity.com/paper/2607.23602/) (2026)
- [ViSA: Visited-State Augmentation for Generalized Goal-Space Contrastive Reinforcement Learning](https://mlsanity.com/paper/2603.14887/) (2026)

## Abstract (arXiv)

Goal-Conditioned Reinforcement Learning (GCRL) can enable agents to spontaneously set diverse goals to learn a set of skills. Despite the excellent works proposed in various fields, reaching distant goals in temporally extended tasks remains a challenge for GCRL. Current works tackled this problem by leveraging planning algorithms to plan intermediate subgoals to augment GCRL. Their methods need two crucial requirements: (i) a state representation space to search valid subgoals, and (ii) a distance function to measure the reachability of subgoals. However, they struggle to scale to high-dimensional state space due to their non-compact representations. Moreover, they cannot collect high-quality training data through standard GC policies, which results in an inaccurate distance function. Both affect the efficiency and performance of planning and policy learning. In the paper, we propose a goal-conditioned RL algorithm combined with Disentanglement-based Reachability Planning (REPlan) to solve temporally extended tasks. In REPlan, a Disentangled Representation Module (DRM) is proposed to learn compact representations which disentangle robot poses and object positions from high-dimensional observations in a self-supervised manner. A simple REachability discrimination Module (REM) is also designed to determine the temporal distance of subgoals. Moreover, REM computes intrinsic bonuses to encourage the collection of novel states for training. We evaluate our REPlan in three vision-based simulation tasks and one real-world task. The experiments demonstrate that our REPlan significantly outperforms the prior state-of-the-art methods in solving temporally extended tasks.