Toward Agents That Reason About Their Computation
On the map: Deep reinforcement learning methods → RL training of language and agent models
Abstract (arXiv)
While reinforcement learning agents can achieve superhuman performance in many complex tasks, they typically do not become more computationally efficient as they improve. In contrast, humans gradually require less cognitive effort as they become more proficient at a task. If agents could reason about their compute as they learn, could they similarly reduce their computation footprint? If they could, we could have more energy efficient agents or free up compute cycles for other processes like planning. In this paper, we experiment with showing agents the cost of their computation and giving them the ability to control when they use compute. We conduct our experiments on the Arcade Learning Environment, and our results demonstrate that with the same training compute budget, agents that reason about their compute perform better on 75% of games. Furthermore, these agents use three times less compute on average. We analyze individual games and show where agents gain these efficiencies. An interactive version of this paper is available at https://adrianorenstein.com/paper/toward-compute.
Key facts. Cited by 3 papers in the map (data through 14 Oct 2026). Placed in RL training of language and agent models, one of 15 directions in Deep reinforcement learning methods. Builds on 18 of its 40 references that are in the map. First posted on arXiv 26 Oct 2025.
Builds on
18 of 40 references.
References · 18
- 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,459
- 2015Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shiftcited by 7,005
- 2019Root Mean Square Layer Normalizationcited by 1,480
- 2017Revisiting the Arcade Learning Environment: Evaluation Protocols and Open Problems for General Agentscited by 269
- 2020Revisiting Fundamentals of Experience Replaycited by 175
- 2023Bigger, Better, Faster: Human-level Atari with human-level efficiencycited by 147
- 2017Eigenoption Discovery through the Deep Successor Representationcited by 86
- 2021The Option Keyboard: Combining Skills in Reinforcement Learningcited by 67
- 2017When Waiting is not an Option : Learning Options with a Deliberation Costcited by 58
- 2017Learning to Repeat: Fine Grained Action Repetition for Deep Reinforcement Learningcited by 39
- 2022Settling the Reward Hypothesiscited by 33
- 2021TempoRL: Learning When to Actcited by 25
6 more
- 2022The Alberta Plan for AI Researchcited by 18
- 2017Learning to select computationscited by 15
- 2024When to Sense and Control? A Time-adaptive Approach for Continuous-Time RLcited by 9
- 2024Beyond The Rainbow: High Performance Deep Reinforcement Learning on a Desktop PCcited by 7
- 2016Dynamic Frame skip Deep Q Networkcited by 5
- 2025OptionZero: Planning with Learned Optionscited by 2
Built on by
Citing papers per year: 2026: 3.
3 papers.
Similar, not cited
Close in wording, no citation either way
- 2026Better, Faster, Stronger: Programmatic Skill Learning Best Reduces Agent Cost
- 2026On the Role of Computation in Reinforcement Learning
- 2024PUZZLES: A Benchmark for Neural Algorithmic Reasoning
- 2026Toward Efficient Agents: Memory, Tool learning, and Planning
- 2026Can AI Agents Learn Their Way to the Top? Evaluating Heuristic Learning in a Long-Running Game Agent Competition
- 2026More Methods, Better Agents? A Reality Check on Self-Distillation in Agentic Reinforcement Learning
- 2026Agent Plasticity: Measuring Self-Improvement Through Experience
- 2025AI Agents as Universal Task Solvers