# Toward Agents That Reason About Their Computation

arXiv: https://arxiv.org/abs/2510.22833 · HTML: https://mlsanity.com/paper/2510.22833/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Placed in Deep RL algorithms and representations, one of 15 directions in Deep reinforcement learning methods.
- Builds on 18 of its 40 references that are in the map.
- First posted on arXiv 26 Oct 2025.

## Placed in

- Deep reinforcement learning methods / [Deep RL algorithms and representations](https://mlsanity.com/deeprl/deep-rl-algorithms-and-representations/)

## Builds on (most cited first, 18 of 18)

- [Chain-of-Thought Prompting Elicits Reasoning in Large Language Models](https://mlsanity.com/paper/2201.11903/) (2022)
- [Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift](https://mlsanity.com/paper/1502.03167/) (2015)
- [Root Mean Square Layer Normalization](https://mlsanity.com/paper/1910.07467/) (2019)
- [Revisiting the Arcade Learning Environment: Evaluation Protocols and Open Problems for General Agents](https://mlsanity.com/paper/1709.06009/) (2017)
- [Revisiting Fundamentals of Experience Replay](https://mlsanity.com/paper/2007.06700/) (2020)
- [Bigger, Better, Faster: Human-level Atari with human-level efficiency](https://mlsanity.com/paper/2305.19452/) (2023)
- [Eigenoption Discovery through the Deep Successor Representation](https://mlsanity.com/paper/1710.11089/) (2017)
- [The Option Keyboard: Combining Skills in Reinforcement Learning](https://mlsanity.com/paper/2106.13105/) (2021)
- [When Waiting is not an Option : Learning Options with a Deliberation Cost](https://mlsanity.com/paper/1709.04571/) (2017)
- [Learning to Repeat: Fine Grained Action Repetition for Deep Reinforcement Learning](https://mlsanity.com/paper/1702.06054/) (2017)
- [Settling the Reward Hypothesis](https://mlsanity.com/paper/2212.10420/) (2022)
- [TempoRL: Learning When to Act](https://mlsanity.com/paper/2106.05262/) (2021)
- [The Alberta Plan for AI Research](https://mlsanity.com/paper/2208.11173/) (2022)
- [Learning to select computations](https://mlsanity.com/paper/1711.06892/) (2017)
- [When to Sense and Control? A Time-adaptive Approach for Continuous-Time RL](https://mlsanity.com/paper/2406.01163/) (2024)
- [Beyond The Rainbow: High Performance Deep Reinforcement Learning on a Desktop PC](https://mlsanity.com/paper/2411.03820/) (2024)
- [Dynamic Frame skip Deep Q Network](https://mlsanity.com/paper/1605.05365/) (2016)
- [OptionZero: Planning with Learned Options](https://mlsanity.com/paper/2502.16634/) (2025)

## Built on by (most cited first, 3 of 3)

- [For How Long Should We Be Punching? Learning Action Duration in Fighting Games](https://mlsanity.com/paper/2605.20911/) (2026)
- [When to Plan: Learning to Select Between Reactive Control and Deliberative Planning](https://mlsanity.com/paper/2607.16421/) (2026)
- [Do Not Imitate, Reinforce: Iterative Classification via Belief Refinement](https://mlsanity.com/paper/2604.22110/) (2026)

## Abstract (arXiv)

While reinforcement learning agents can achieve superhuman performance in many complex tasks, they typically do not become more computationally efficient as they improve. In contrast, humans gradually require less cognitive effort as they become more proficient at a task. If agents could reason about their compute as they learn, could they similarly reduce their computation footprint? If they could, we could have more energy efficient agents or free up compute cycles for other processes like planning. In this paper, we experiment with showing agents the cost of their computation and giving them the ability to control when they use compute. We conduct our experiments on the Arcade Learning Environment, and our results demonstrate that with the same training compute budget, agents that reason about their compute perform better on 75% of games. Furthermore, these agents use three times less compute on average. We analyze individual games and show where agents gain these efficiencies. An interactive version of this paper is available at https://adrianorenstein.com/paper/toward-compute.