Finding the Time to Think: Learning Planning Budgets in Real-Time RL
Not on the map. Why?
Abstract (arXiv)
Deliberating takes time. In real-time settings, that time is not free. Standard reinforcement learning (RL) sidesteps this as the environment waits indefinitely for the agent's decision. Instead, we study real-time RL environments where the environment progresses while waiting for the agent's action. Building on prior real-time formalizations, we introduce variable-delay real-time RL, where the agent chooses how long to deliberate at each decision point since the environment progresses. For the planning agents we use, the right delay is state-dependent, and naively planning how long to plan can paralyze the agent. We instead approach this setting by training a lightweight gating policy on top of a planner to select state-dependent planning budgets. Across real-time Pac-Man, Tetris, Snake, Speed Hex, and Speed Go, our gating policy outperforms fixed-budget and heuristic baselines, and transfers to a real-time setup where the environment and agent run on two different GPUs.
Key facts. Cited by 2 papers in the map (data through 7 Oct 2026). Builds on 36 of its 68 references that are in the map. First posted on arXiv 24 Jun 2026.
Builds on
36 of 68 references.
References · 36
- 2017Proximal Policy Optimization Algorithmscited by 14,441
- 2024Scaling LLM Test-Time Compute Optimally can be More Effective than Scaling Model Parameterscited by 1,928
- 2015High-Dimensional Continuous Control Using Generalized Advantage Estimationcited by 1,916
- 2019Mastering Atari, Go, Chess and Shogi by Planning with a Learned Modelcited by 1,327
- 2025s1: Simple test-time scalingcited by 1,311
- 2023FrugalGPT: How to Use Large Language Models While Reducing Cost and Improving Performancecited by 791
- 2024RouteLLM: Learning to Route LLMs with Preference Datacited by 633
- 2016Adaptive Computation Time for Recurrent Neural Networkscited by 479
- 2016The Option-Critic Architecturecited by 433
- 2025L1: Controlling How Long A Reasoning Model Thinks With Reinforcement Learningcited by 381
- 2022Confident Adaptive Language Modelingcited by 297
- 2023Speculative Decoding with Big Little Decodercited by 163
24 more
- 2021PonderNet: Learning to Pondercited by 156
- 2025DAST: Difficulty-Adaptive Slow-Thinking for Large Reasoning Modelscited by 154
- 2017Imagination-Augmented Agents for Deep Reinforcement Learningcited by 137
- 2025Thinkless: LLM Learns When to Thinkcited by 107
- 2020On the role of planning in model-based deep reinforcement learningcited by 57
- 2020Reinforcement Learning with Random Delayscited by 56
- 2023Pgx: Hardware-Accelerated Parallel Game Simulators for Reinforcement Learningcited by 45
- 2019Real-Time Reinforcement Learningcited by 38
- 2018Learning to Search with MCTSnetscited by 32
- 2020Thinking While Moving: Deep Reinforcement Learning with Concurrent Controlcited by 29
- 2021Acting in Delayed Environments with Non-Stationary Markov Policiescited by 28
- 2019Combining Q-Learning and Search with Amortized Value Estimatescited by 19
- 2024Fast TRAC: A Parameter-Free Optimizer for Lifelong Reinforcement Learningcited by 16
- 2017Learning to select computationscited by 15
- 2023Thinker: Learning to Plan and Actcited by 13
- 2018Reactive Reinforcement Learning in Asynchronous Environmentscited by 12
- 2017Metacontrol for Adaptive Imagination-Based Optimizationcited by 8
- 2024Enabling Realtime Reinforcement Learning at Scale with Staggered Asynchronous Inferencecited by 7
- 2025Handling Delay in Real-Time Reinforcement Learningcited by 7
- 2020Static and Dynamic Values of Computation in MCTScited by 5
- 2015Metareasoning for Planning Under Uncertaintycited by 3
- 2026Predictive Scheduling for Efficient Inference-Time Reasoning in Large Language Modelscited by 2
- 2013Reasoning, Metareasoning, and Mathematical Truth: Studies of Theorem Proving under Limited Resourcescited by 1
- 2024Permutation Invariant Learning with High-Dimensional Particle Filterscited by 1
Built on by
Citing papers per year: 2026: 2.
2 papers.