# Learning model-based planning from scratch

arXiv: https://arxiv.org/abs/1707.06170 · HTML: https://mlsanity.com/paper/1707.06170/

## Key facts

- Cited by 22 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 19 Jul 2017.

## Built on by (most cited first, 22 of 22)

- [Is Your LLM Secretly a World Model of the Internet? Model-Based Planning for Web Agents](https://mlsanity.com/paper/2411.06559/) (2024)
- [Muesli: Combining Improvements in Policy Optimization](https://mlsanity.com/paper/2104.06159/) (2021)
- [Retrieval-Augmented Reinforcement Learning](https://mlsanity.com/paper/2202.08417/) (2022)
- [MBRL-Lib: A Modular Library for Model-based Reinforcement Learning](https://mlsanity.com/paper/2104.10159/) (2021)
- [ViMo: A Generative Visual GUI World Model for App Agents](https://mlsanity.com/paper/2504.13936/) (2025)
- [High-Accuracy Model-Based Reinforcement Learning, a Survey](https://mlsanity.com/paper/2107.08241/) (2021)
- [Thinker: Learning to Plan and Act](https://mlsanity.com/paper/2307.14993/) (2023)
- [Learning Graph Search Heuristics](https://mlsanity.com/paper/2212.03978/) (2022)
- [Policy Gradients Incorporating the Future](https://mlsanity.com/paper/2108.02096/) (2021)
- [Computing a human-like reaction time metric from stable recurrent vision models](https://mlsanity.com/paper/2306.11582/) (2023)
- [From Myopic Selection to Long-Horizon Awareness: Sequential LLM Routing for Multi-Turn Dialogue](https://mlsanity.com/paper/2604.12385/) (2026)
- [Learning Abstractions for Hierarchical Planning in Program-Synthesis Agents](https://mlsanity.com/paper/2602.00929/) (2026)
- [ExPoSe: Combining State-Based Exploration with Gradient-Based Online Search](https://mlsanity.com/paper/2202.01461/) (2022)
- [A Review of Symbolic, Subsymbolic and Hybrid Methods for Sequential Decision Making](https://mlsanity.com/paper/2304.10590/) (2023)
- [Beyond Reactive Safety: Risk-Aware LLM Alignment via Long-Horizon Simulation](https://mlsanity.com/paper/2506.20949/) (2025)
- [Reinforced Planning with Latent World Models](https://mlsanity.com/paper/2608.18669/) (2026)
- [Generic Itemset Mining Based on Reinforcement Learning](https://mlsanity.com/paper/2105.07753/) (2021)
- [Differentiable Tree Search Network](https://mlsanity.com/paper/2401.11660/) (2024)
- [World Models as an Intermediary between Agents and the Real World](https://mlsanity.com/paper/2602.00785/) (2026)
- [Deep imagination is a close to optimal policy for planning in large decision trees under limited resources](https://mlsanity.com/paper/2104.06339/) (2021)
- [Thinking agents for zero-shot generalization to qualitatively novel tasks](https://mlsanity.com/paper/2503.19815/) (2025)
- [Learning to Play: Reinforcement Learning and Games](https://mlsanity.com/paper/2610.17391/) (2026)

## Abstract (arXiv)

Conventional wisdom holds that model-based planning is a powerful approach to sequential decision-making. It is often very challenging in practice, however, because while a model can be used to evaluate a plan, it does not prescribe how to construct a plan. Here we introduce the "Imagination-based Planner", the first model-based, sequential decision-making agent that can learn to construct, evaluate, and execute plans. Before any action, it can perform a variable number of imagination steps, which involve proposing an imagined action and evaluating it with its model-based imagination. All imagined actions and outcomes are aggregated, iteratively, into a "plan context" which conditions future real and imagined actions. The agent can even decide how to imagine: testing out alternative imagined actions, chaining sequences of actions together, or building a more complex "imagination tree" by navigating flexibly among the previously imagined states using a learned policy. And our agent can learn to plan economically, jointly optimizing for external rewards and computational costs associated with using its imagination. We show that our architecture can learn to solve a challenging continuous control problem, and also learn elaborate planning strategies in a discrete maze-solving task. Our work opens a new direction toward learning the components of a model-based planning system and how to use them.