# When to Plan: Learning to Select Between Reactive Control and Deliberative Planning

arXiv: https://arxiv.org/abs/2607.16421 · HTML: https://mlsanity.com/paper/2607.16421/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 26 of its 56 references that are in the map.
- First posted on arXiv 17 Jul 2026.

## Builds on (most cited first, 26 of 26)

- [Proximal Policy Optimization Algorithms](https://mlsanity.com/paper/1707.06347/) (2017)
- [Simple and Scalable Predictive Uncertainty Estimation using Deep Ensembles](https://mlsanity.com/paper/1612.01474/) (2016)
- [RouteLLM: Learning to Route LLMs with Preference Data](https://mlsanity.com/paper/2406.18665/) (2024)
- [Minigrid & Miniworld: Modular & Customizable Reinforcement Learning Environments for Goal-Oriented Tasks](https://mlsanity.com/paper/2306.13831/) (2023)
- [Thinking Fast and Slow with Deep Learning and Tree Search](https://mlsanity.com/paper/1705.08439/) (2017)
- [Learning Agile Soccer Skills for a Bipedal Robot with Deep Reinforcement Learning](https://mlsanity.com/paper/2304.13653/) (2023)
- [Thinkless: LLM Learns When to Think](https://mlsanity.com/paper/2505.13379/) (2025)
- [Learning to Repeat: Fine Grained Action Repetition for Deep Reinforcement Learning](https://mlsanity.com/paper/1702.06054/) (2017)
- [Control Frequency Adaptation via Action Persistence in Batch Reinforcement Learning](https://mlsanity.com/paper/2002.06836/) (2020)
- [Rational Metareasoning for Large Language Models](https://mlsanity.com/paper/2410.05563/) (2024)
- [Learning When to Plan: Efficiently Allocating Test-Time Compute for LLM Agents](https://mlsanity.com/paper/2509.03581/) (2025)
- [The Benefits of Model-Based Generalization in Reinforcement Learning](https://mlsanity.com/paper/2211.02222/) (2022)
- [Learning to select computations](https://mlsanity.com/paper/1711.06892/) (2017)
- [Rethinking Social Robot Navigation: Leveraging the Best of Two Worlds](https://mlsanity.com/paper/2309.13466/) (2023)
- [Dynamic Decision Frequency with Continuous Options](https://mlsanity.com/paper/2212.04407/) (2022)
- [Thinker: Learning to Think Fast and Slow](https://mlsanity.com/paper/2505.21097/) (2025)
- [Metacontrol for Adaptive Imagination-Based Optimization](https://mlsanity.com/paper/1705.02670/) (2017)
- [All-in-One: A DRL-based Control Switch Combining State-of-the-art Navigation Planners](https://mlsanity.com/paper/2109.11636/) (2021)
- [When Can Model-Free Reinforcement Learning be Enough for Thinking?](https://mlsanity.com/paper/2506.17124/) (2025)
- [Zero-Overhead Introspection for Adaptive Test-Time Compute](https://mlsanity.com/paper/2512.01457/) (2025)
- [When to Replan? An Adaptive Replanning Strategy for Autonomous Navigation using Deep Reinforcement Learning](https://mlsanity.com/paper/2304.12046/) (2023)
- [Toward Agents That Reason About Their Computation](https://mlsanity.com/paper/2510.22833/) (2025)
- [Learning-based Event-triggered MPC with Gaussian processes under terminal constraints](https://mlsanity.com/paper/2110.12214/) (2021)
- [Learning whom to trust in navigation: dynamically switching between classical and neural planning](https://mlsanity.com/paper/2307.16710/) (2023)
- [Hybrid Classical/RL Local Planner for Ground Robot Navigation](https://mlsanity.com/paper/2410.03066/) (2024)
- [A Novel Model for Arbitration between Planning and Habitual Control Systems](https://mlsanity.com/paper/1712.02441/) (2017)

## Built on by (most cited first, 1 of 1)

- [Think Fast, Plan Selectively: Adaptive Deliberation for Efficient Data-Driven MPC](https://mlsanity.com/paper/2609.32591/) (2026)

## Abstract (arXiv)

It has long been recognized that humans have the ability to switch between fast, reactive decision-making and slower, deliberative planning. In this paper, we study the question of how to learn this ability, known as meta-reasoning, in artificial agents. We model reactive decision-making as a policy that directly maps state observations to actions. Such policies can be trained with reinforcement learning (RL) or imitation learning, but may generalize poorly outside of their training distribution. Alternatively, model-based decision-time planning is more likely to produce good actions across a broader set of states but requires additional computation time, which delays acting. In this work, we introduce an RL method for training a meta-reasoning policy that allocates computation by conditioning on a reactive-policy uncertainty score. This score enables it to predict when the reactive policy is likely to perform poorly and when planning is needed. We conduct an empirical study on motion planning and navigation environments, showing that this design enables the meta-reasoning policy to learn when the reactive policy provides a good-enough action versus when decision-time planning is needed. Additionally, we show that our design enables the meta-agent to shift toward fully reactive control as the reactive policy improves.