# Think Too Fast Nor Too Slow: The Computational Trade-off Between Planning And Reinforcement Learning

arXiv: https://arxiv.org/abs/2005.07404 · HTML: https://mlsanity.com/paper/2005.07404/

## Key facts

- Cited by 5 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 15 May 2020.

## Built on by (most cited first, 5 of 5)

- [Improve Agents without Retraining: Parallel Tree Search with Off-Policy Correction](https://mlsanity.com/paper/2107.01715/) (2021)
- [Planning and Learning with Adaptive Lookahead](https://mlsanity.com/paper/2201.12403/) (2022)
- [Multi-Bellman operator for convergence of Q-learning with linear function approximation](https://mlsanity.com/paper/2309.16819/) (2023)
- [Counting Hours, Counting Losses: The Toll of Unpredictable Work Schedules on Financial Security](https://mlsanity.com/paper/2504.07719/) (2025)
- [Effective Online 3D Bin Packing with Lookahead Parcels Using Monte Carlo Tree Search](https://mlsanity.com/paper/2601.02649/) (2026)

## Abstract (arXiv)

Planning and reinforcement learning are two key approaches to sequential decision making. Multi-step approximate real-time dynamic programming, a recently successful algorithm class of which AlphaZero [Silver et al., 2018] is an example, combines both by nesting planning within a learning loop. However, the combination of planning and learning introduces a new question: how should we balance time spend on planning, learning and acting? The importance of this trade-off has not been explicitly studied before. We show that it is actually of key importance, with computational results indicating that we should neither plan too long nor too short. Conceptually, we identify a new spectrum of planning-learning algorithms which ranges from exhaustive search (long planning) to model-free RL (no planning), with optimal performance achieved midway.