mlsanityData to 14 Oct 2026

On the Role of Computation in Reinforcement Learning

arXiv 2602.05999 ↗First posted 5 Feb 2026cs.LGcited by 3

On the map: Deep reinforcement learning methods → Deep RL algorithms and representations

Abstract (arXiv)

How does the amount of compute available to a reinforcement learning (RL) policy affect its learning? Can policies using a fixed amount of parameters, still benefit from additional compute? The standard RL framework does not provide a language to answer these questions formally. Empirically, deep RL policies are often parameterized as neural networks with static architectures, conflating the amount of compute and the number of parameters. In this paper, we formalize compute bounded policies and prove that policies which use more compute can solve problems and generalize to longer-horizon tasks that are outside the scope of policies with less compute. Building on prior work in algorithmic learning and model-free planning, we propose a minimal architecture that can use a variable amount of compute. Our experiments complement our theory. On a set 31 different tasks spanning online and offline RL, we show that (1) this architecture achieves stronger performance simply by using more compute, and (2) stronger generalization on longer-horizon test tasks compared to standard feedforward networks or deep residual network using up to 5 times more parameters.

Key facts. Cited by 3 papers in the map (data through 14 Oct 2026). Placed in Deep RL algorithms and representations, one of 15 directions in Deep reinforcement learning methods. Builds on 46 of its 66 references that are in the map. First posted on arXiv 5 Feb 2026.

Builds on

46 of 66 references.

Closest ancestors · 3

Same direction, newest first

  1. 2025When Can Model-Free Reinforcement Learning be Enough for Thinking?20 Jun
  2. 20251000 Layer Networks for Self-Supervised RL: Scaling Depth Can Enable New Goal-Reaching Capabilities19 Mar
  3. 2025Horizon Generalization in Reinforcement Learning6 Jan

Other references · 43

  1. 2015Deep Residual Learning for Image Recognitioncited by 52,096
  2. 2020Language Models are Few-Shot Learnerscited by 32,419
  3. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,459
  4. 2021Training Verifiers to Solve Math Word Problemscited by 10,109
  5. 2016Layer Normalizationcited by 5,669
  6. 2021The Power of Scale for Parameter-Efficient Prompt Tuningcited by 3,795
  7. 2013Playing Atari with Deep Reinforcement Learningcited by 3,274
  8. 2023Tree of Thoughts: Deliberate Problem Solving with Large Language Modelscited by 3,211
  9. 2022Emergent Abilities of Large Language Modelscited by 2,512
  10. 2015Convolutional LSTM Network: A Machine Learning Approach for Precipitation Nowcastingcited by 1,436
  11. 2019Mastering Atari, Go, Chess and Shogi by Planning with a Learned Modelcited by 1,353
  12. 2014On the Properties of Neural Machine Translation: Encoder-Decoder Approachescited by 1,304
31 more
  1. 2021Offline Reinforcement Learning with Implicit Q-Learningcited by 1,293
  2. 2018Learning Latent Dynamics for Planning from Pixelscited by 1,264
  3. 2019When to Trust Your Model: Model-Based Policy Optimizationcited by 773
  4. 2019Solving Rubik's Cube with a Robot Handcited by 734
  5. 2014Neural Turing Machinescited by 644
  6. 2023Language Agent Tree Search Unifies Reasoning Acting and Planning in Language Modelscited by 574
  7. 2016Adaptive Computation Time for Recurrent Neural Networkscited by 506
  8. 2025Scaling up Test-Time Compute with Latent Reasoning: A Recurrent Depth Approachcited by 415
  9. 2022Large Language Models are Better Reasoners with Self-Verificationcited by 307
  10. 2022Contrastive Learning as Goal-Conditioned Reinforcement Learningcited by 253
  11. 2024OGBench: Benchmarking Offline Goal-Conditioned RLcited by 238
  12. 2023What Algorithms can Transformers Learn? A Study in Length Generalizationcited by 199
  13. 2015Learning Continuous Control Policies by Stochastic Value Gradientscited by 162
  14. 2016Value Iteration Networkscited by 160
  15. 2023Looped Transformers are Better at Learning Learning Algorithmscited by 149
  16. 2023What Formal Languages Can Transformers Express? A Surveycited by 124
  17. 2021Can You Learn an Algorithm? Generalizing from Easy to Hard Problems with Recurrent Networkscited by 117
  18. 2020The Value Equivalence Principle for Model-Based Reinforcement Learningcited by 94
  19. 2016The Predictron: End-To-End Learning and Planningcited by 93
  20. 2024Transformers Can Achieve Length Generalization But Not Robustlycited by 86
  21. 2019An investigation of model-free planningcited by 66
  22. 2021The Difficulty of Passive Learning in Deep Reinforcement Learningcited by 50
  23. 2021Mismatched No More: Joint Model-Policy Optimization for Model-Based RLcited by 36
  24. 2024Learning Iterative Reasoning through Energy Diffusioncited by 36
  25. 2018Learning to Search with MCTSnetscited by 33
  26. 2025Self-Improving Transformers Overcome Easy-to-Hard and Length Generalization Challengescited by 31
  27. 2023DexPBT: Scaling up Dexterous Manipulation for Hand-Arm Systems with Population Based Trainingcited by 29
  28. 2022Simplifying Model-based RL: Learning Representations, Latent-space Models, and Policies with One Objectivecited by 28
  29. 2024Position Coupling: Improving Length Generalization of Arithmetic Transformers Using Task Structurecited by 23
  30. 2025Interpreting Emergent Planning in Model-Free Reinforcement Learningcited by 15
  31. 2025Is Temporal Difference Learning the Gold Standard for Stitching in RL?cited by 4

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 3.

3 papers.

  1. 2026Solve the Loop: Attractor Models for Language and Reasoningcited by 13
  2. 2026FlexLoop: Depth-Elastic Looped Policies for Adaptive Test-Time Computation in Deep RLcited by 0
  3. 2026Looped Actor: Depth-Recurrent Reasoning Models for Reinforcement Learningcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026Where Should RL Post-Training Compute Go? Model Size, Search, Learning, and Feedback
  2. 2026Sample-Efficient Neurosymbolic Deep Reinforcement Learning
  3. 2024Efficient Exploration in Deep Reinforcement Learning: A Novel Bayesian Actor-Critic Algorithm
  4. 2024A Comprehensive Survey of Reinforcement Learning: From Algorithms to Practical Challenges
  5. 2025Rule-Guided Reinforcement Learning Policy Evaluation and Improvement
  6. 2025Towards General-Purpose Model-Free Reinforcement Learning
  7. 2025Context-Sensitive Abstractions for Reinforcement Learning with Parameterized Actions
  8. 2021On Multi-objective Policy Optimization as a Tool for Reinforcement Learning: Case Studies in Offline RL and Finetuning