# Learning When to Plan: Efficiently Allocating Test-Time Compute for LLM Agents

arXiv: https://arxiv.org/abs/2509.03581 · HTML: https://mlsanity.com/paper/2509.03581/

## Key facts

- Cited by 20 papers in the map (data through 14 Oct 2026).
- Placed in Reasoning for agents, planning and code, one of 17 directions in LLM reasoning.
- Builds on 55 of its 69 references that are in the map.
- First posted on arXiv 3 Sep 2025.

## Placed in

- LLM reasoning / [Reasoning for agents, planning and code](https://mlsanity.com/lr/reasoning-for-agents-planning-and-code/)
- LLM agents and tool use / [General LLM agent frameworks and benchmarks](https://mlsanity.com/agents/general-llm-agent-frameworks-and-benchmarks/)

## Builds on (most cited first, 30 of 55)

- [Proximal Policy Optimization Algorithms](https://mlsanity.com/paper/1707.06347/) (2017)
- [The Llama 3 Herd of Models](https://mlsanity.com/paper/2407.21783/) (2024)
- [LoRA: Low-Rank Adaptation of Large Language Models](https://mlsanity.com/paper/2106.09685/) (2021)
- [Chain-of-Thought Prompting Elicits Reasoning in Large Language Models](https://mlsanity.com/paper/2201.11903/) (2022)
- [Training Verifiers to Solve Math Word Problems](https://mlsanity.com/paper/2110.14168/) (2021)
- [ReAct: Synergizing Reasoning and Acting in Language Models](https://mlsanity.com/paper/2210.03629/) (2022)
- [Efficient Memory Management for Large Language Model Serving with PagedAttention](https://mlsanity.com/paper/2309.06180/) (2023)
- [Self-Consistency Improves Chain of Thought Reasoning in Language Models](https://mlsanity.com/paper/2203.11171/) (2022)
- [Measuring Mathematical Problem Solving With the MATH Dataset](https://mlsanity.com/paper/2103.03874/) (2021)
- [Reflexion: Language Agents with Verbal Reinforcement Learning](https://mlsanity.com/paper/2303.11366/) (2023)
- [Gemini 2.5: Pushing the Frontier with Advanced Reasoning, Multimodality, Long Context, and Next Generation Agentic Capabilities](https://mlsanity.com/paper/2507.06261/) (2025)
- [Lost in the Middle: How Language Models Use Long Contexts](https://mlsanity.com/paper/2307.03172/) (2023)
- [Tree of Thoughts: Deliberate Problem Solving with Large Language Models](https://mlsanity.com/paper/2305.10601/) (2023)
- [Do As I Can, Not As I Say: Grounding Language in Robotic Affordances](https://mlsanity.com/paper/2204.01691/) (2022)
- [Voyager: An Open-Ended Embodied Agent with Large Language Models](https://mlsanity.com/paper/2305.16291/) (2023)
- [Dream to Control: Learning Behaviors by Latent Imagination](https://mlsanity.com/paper/1912.01603/) (2019)
- [World Models](https://mlsanity.com/paper/1803.10122/) (2018)
- [Mastering Atari, Go, Chess and Shogi by Planning with a Learned Model](https://mlsanity.com/paper/1911.08265/) (2019)
- [s1: Simple test-time scaling](https://mlsanity.com/paper/2501.19393/) (2025)
- [Learning Latent Dynamics for Planning from Pixels](https://mlsanity.com/paper/1811.04551/) (2018)
- [Tulu 3: Pushing Frontiers in Open Language Model Post-Training](https://mlsanity.com/paper/2411.15124/) (2024)
- [STaR: Bootstrapping Reasoning With Reasoning](https://mlsanity.com/paper/2203.14465/) (2022)
- [Graph of Thoughts: Solving Elaborate Problems with Large Language Models](https://mlsanity.com/paper/2308.09687/) (2023)
- [Reasoning with Language Model is Planning with World Model](https://mlsanity.com/paper/2305.14992/) (2023)
- [Mastering Chess and Shogi by Self-Play with a General Reinforcement Learning Algorithm](https://mlsanity.com/paper/1712.01815/) (2017)
- [Promptbreeder: Self-Referential Self-Improvement Via Prompt Evolution](https://mlsanity.com/paper/2309.16797/) (2023)
- [Ray: A Distributed Framework for Emerging AI Applications](https://mlsanity.com/paper/1712.05889/) (2017)
- [Stop Overthinking: A Survey on Efficient Reasoning for Large Language Models](https://mlsanity.com/paper/2503.16419/) (2025)
- [The Option-Critic Architecture](https://mlsanity.com/paper/1609.05140/) (2016)
- [Training Language Models to Self-Correct via Reinforcement Learning](https://mlsanity.com/paper/2409.12917/) (2024)

## Built on by (most cited first, 20 of 20)

- [Toward Efficient Agents: Memory, Tool learning, and Planning](https://mlsanity.com/paper/2601.14192/) (2026)
- [Same Signal, Opposite Meaning: Direction-Informed Adaptive Learning for LLM Agents](https://mlsanity.com/paper/2605.06908/) (2026)
- [Tycho: Active Abstraction with Programmatic World Models for ARC-AGI-3](https://mlsanity.com/paper/2607.28287/) (2026)
- [Timely Machine: Awareness of Time Makes Test-Time Scaling Agentic](https://mlsanity.com/paper/2601.16486/) (2026)
- [Re-FORC: Adaptive Reward Prediction for Efficient Chain-of-Thought Reasoning](https://mlsanity.com/paper/2511.02130/) (2025)
- [What If We Allocate Test-Time Compute Adaptively?](https://mlsanity.com/paper/2602.01070/) (2026)
- [Can LLMs Perceive Time? An Empirical Investigation](https://mlsanity.com/paper/2604.00010/) (2026)
- [A Survey on Self-Improving Test-Time Intelligence: Feedback-Driven Adapting, Learning, and Scaling at Inference](https://mlsanity.com/paper/2609.01679/) (2026)
- [TodoEvolve: Learning to Architect Agent Planning Systems](https://mlsanity.com/paper/2602.07839/) (2026)
- [PaT: Planning-after-Trial for Efficient Test-Time Code Generation](https://mlsanity.com/paper/2605.07248/) (2026)
- [LLM-WikiRace Benchmark: How Far Can LLMs Plan over Real-World Knowledge Graphs?](https://mlsanity.com/paper/2602.16902/) (2026)
- [From Skills to Talent: Organising Heterogeneous Agents as a Real-World Company](https://mlsanity.com/paper/2604.22446/) (2026)
- [Rethinking Agentic Reinforcement Learning In Large Language Models](https://mlsanity.com/paper/2604.27859/) (2026)
- [When to Plan: Learning to Select Between Reactive Control and Deliberative Planning](https://mlsanity.com/paper/2607.16421/) (2026)
- [Mint-Agent: Introducing Finance-Native Agentic Foundation Models](https://mlsanity.com/paper/2608.16386/) (2026)
- [Elastic Horizon: Discovering the Effective Interaction Frontier in Agentic Reinforcement Learning](https://mlsanity.com/paper/2609.07247/) (2026)
- [Consistent Plan-Act for Long-Horizon Agentic Tasks](https://mlsanity.com/paper/2609.38891/) (2026)
- [Beyond Prediction: Steering VLM Agents with Retrospective World Modeling](https://mlsanity.com/paper/2609.39101/) (2026)
- [ForkPilot: Self-Evolving Policy for Retrospective Search in Long-Horizon Agents](https://mlsanity.com/paper/2610.04889/) (2026)
- [On the Clock: Towards Punctual and Productive Time-Budgeted AI Agents](https://mlsanity.com/paper/2610.10833/) (2026)

## Abstract (arXiv)

Training large language models (LLMs) to reason via reinforcement learning (RL) significantly improves their problem-solving capabilities. In agentic settings, existing methods like ReAct prompt LLMs to explicitly plan before every action; however, we demonstrate that always planning is computationally expensive and degrades performance on long-horizon tasks, while never planning further limits performance. To address this, we introduce a conceptual framework formalizing dynamic planning for LLM agents, enabling them to flexibly decide when to allocate test-time compute for planning. We propose a simple two-stage training pipeline: (1) supervised fine-tuning on diverse synthetic data to prime models for dynamic planning, and (2) RL to refine this capability in long-horizon environments. Experiments on the Crafter environment show that dynamic planning agents trained with this approach are more sample-efficient and consistently achieve more complex objectives. Additionally, we demonstrate that these agents can be effectively steered by human-written plans, surpassing their independent capabilities and highlighting the potential for safer and more collaborative agentic systems.