# World Model as a Graph: Learning Latent Landmarks for Planning

arXiv: https://arxiv.org/abs/2011.12491 · HTML: https://mlsanity.com/paper/2011.12491/

## Key facts

- Cited by 69 papers in the map (data through 14 Oct 2026).
- Placed in Latent predictive world models, one of 11 directions in World models.
- Builds on 8 of its 53 references that are in the map.
- First posted on arXiv 25 Nov 2020.

## Placed in

- World models / [Latent predictive world models](https://mlsanity.com/wm/latent-predictive-world-models/)

## Builds on (most cited first, 8 of 8)

- [Learning Latent Dynamics for Planning from Pixels](https://mlsanity.com/paper/1811.04551/) (2018)
- [Mastering Atari with Discrete World Models](https://mlsanity.com/paper/2010.02193/) (2020)
- [Recurrent World Models Facilitate Policy Evolution](https://mlsanity.com/paper/1809.01999/) (2018)
- [When to Trust Your Model: Model-Based Policy Optimization](https://mlsanity.com/paper/1906.08253/) (2019)
- [Neural Network Dynamics for Model-Based Deep Reinforcement Learning with Model-Free Fine-Tuning](https://mlsanity.com/paper/1708.02596/) (2017)
- [Model-Ensemble Trust-Region Policy Optimization](https://mlsanity.com/paper/1802.10592/) (2018)
- [Algorithmic Framework for Model-based Deep Reinforcement Learning with Theoretical Guarantees](https://mlsanity.com/paper/1807.03858/) (2018)
- [Exploring Model-based Planning with Policy Networks](https://mlsanity.com/paper/1906.08649/) (2019)

## Built on by (most cited first, 30 of 69)

- [Goal-Conditioned Reinforcement Learning: Problems and Solutions](https://mlsanity.com/paper/2201.08299/) (2022)
- [HIQL: Offline Goal-Conditioned RL with Latent States as Actions](https://mlsanity.com/paper/2307.11949/) (2023)
- [Goal-Conditioned Reinforcement Learning with Imagined Subgoals](https://mlsanity.com/paper/2107.00541/) (2021)
- [Rapid Exploration for Open-World Navigation with Latent Goal Models](https://mlsanity.com/paper/2104.05859/) (2021)
- [World Models for Autonomous Driving: An Initial Survey](https://mlsanity.com/paper/2403.02622/) (2024)
- [Landmark-Guided Subgoal Generation in Hierarchical Reinforcement Learning](https://mlsanity.com/paper/2110.13625/) (2021)
- [Universal Visual Decomposer: Long-Horizon Manipulation Made Easy](https://mlsanity.com/paper/2310.08581/) (2023)
- [Embodied AI: From LLMs to World Models](https://mlsanity.com/paper/2509.20021/) (2025)
- [Option-aware Temporally Abstracted Value for Offline Goal-Conditioned Reinforcement Learning](https://mlsanity.com/paper/2505.12737/) (2025)
- [Making Large Language Models into World Models with Precondition and Effect Knowledge](https://mlsanity.com/paper/2409.12278/) (2024)
- [Subgoal Search For Complex Reasoning Tasks](https://mlsanity.com/paper/2108.11204/) (2021)
- [DHRL: A Graph-Based Approach for Long-Horizon and Sparse Hierarchical Reinforcement Learning](https://mlsanity.com/paper/2210.05150/) (2022)
- [Topological Experience Replay](https://mlsanity.com/paper/2203.15845/) (2022)
- [Graph World Model](https://mlsanity.com/paper/2507.10539/) (2025)
- [Explainable Autonomous Robots: A Survey and Perspective](https://mlsanity.com/paper/2105.02658/) (2021)
- [GHIL-Glue: Hierarchical Control with Filtered Subgoal Images](https://mlsanity.com/paper/2410.20018/) (2024)
- [Interaction Modeling with Multiplex Attention](https://mlsanity.com/paper/2208.10660/) (2022)
- [Offline Imitation Learning Through Graph Search and Retrieval](https://mlsanity.com/paper/2407.15403/) (2024)
- [Imitating Graph-Based Planning with Goal-Conditioned Policies](https://mlsanity.com/paper/2303.11166/) (2023)
- [Learning Domain Invariant Representations in Goal-conditioned Block MDPs](https://mlsanity.com/paper/2110.14248/) (2021)
- [Fast and Precise: Adjusting Planning Horizon with Adaptive Subgoal Search](https://mlsanity.com/paper/2206.00702/) (2022)
- [Goal-Space Planning with Subgoal Models](https://mlsanity.com/paper/2206.02902/) (2022)
- [Value Memory Graph: A Graph-Structured World Model for Offline Reinforcement Learning](https://mlsanity.com/paper/2206.04384/) (2022)
- [Discrete Factorial Representations as an Abstraction for Goal Conditioned Reinforcement Learning](https://mlsanity.com/paper/2211.00247/) (2022)
- [CQM: Curriculum Reinforcement Learning with a Quantized World Model](https://mlsanity.com/paper/2310.17330/) (2023)
- [Flattening Hierarchies with Policy Bootstrapping](https://mlsanity.com/paper/2505.14975/) (2025)
- [Towards Generalist Robots: A Promising Paradigm via Generative Simulation](https://mlsanity.com/paper/2305.10455/) (2023)
- [Test-Time Graph Search for Goal-Conditioned Reinforcement Learning](https://mlsanity.com/paper/2510.07257/) (2025)
- [DisTop: Discovering a Topological representation to learn diverse and rewarding skills](https://mlsanity.com/paper/2106.03853/) (2021)
- [Graph schemas as abstractions for transfer learning, inference, and planning](https://mlsanity.com/paper/2302.07350/) (2023)

## Abstract (arXiv)

Planning - the ability to analyze the structure of a problem in the large and decompose it into interrelated subproblems - is a hallmark of human intelligence. While deep reinforcement learning (RL) has shown great promise for solving relatively straightforward control tasks, it remains an open problem how to best incorporate planning into existing deep RL paradigms to handle increasingly complex environments. One prominent framework, Model-Based RL, learns a world model and plans using step-by-step virtual rollouts. This type of world model quickly diverges from reality when the planning horizon increases, thus struggling at long-horizon planning. How can we learn world models that endow agents with the ability to do temporally extended reasoning? In this work, we propose to learn graph-structured world models composed of sparse, multi-step transitions. We devise a novel algorithm to learn latent landmarks that are scattered (in terms of reachability) across the goal space as the nodes on the graph. In this same graph, the edges are the reachability estimates distilled from Q-functions. On a variety of high-dimensional continuous control tasks ranging from robotic manipulation to navigation, we demonstrate that our method, named L3P, significantly outperforms prior work, and is oftentimes the only method capable of leveraging both the robustness of model-free RL and generalization of graph-search algorithms. We believe our work is an important step towards scalable planning in reinforcement learning.