mlsanityData to 7 Oct 2026

HRM-Agent: Training a recurrent reasoning model in dynamic environments using reinforcement learning

arXiv 2510.22832 ↗First posted 26 Oct 2025cs.AI cs.LG stat.MLcited by 0

On the map: LLM reasoning → Reasoning for agents, planning and code

Abstract (arXiv)

The Hierarchical Reasoning Model (HRM) has impressive reasoning abilities given its small size, but has only been applied to supervised, static, fully-observable problems. One of HRM's strengths is its ability to adapt its computational effort to the difficulty of the problem. However, in its current form it cannot integrate and reuse computation from previous time-steps if the problem is dynamic, uncertain or partially observable, or be applied where the correct action is undefined, characteristics of many real-world problems. This paper presents HRM-Agent, a variant of HRM trained using only reinforcement learning. We show that HRM can learn to navigate to goals in dynamic and uncertain maze environments. Recent work suggests that HRM's reasoning abilities stem from its recurrent inference process. We explore the dynamics of the recurrent inference process and find evidence that it is successfully reusing computation from earlier environment time-steps.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Placed in Reasoning for agents, planning and code, one of 17 directions in LLM reasoning. Builds on 20 of its 40 references that are in the map. First posted on arXiv 26 Oct 2025.

Builds on

20 of 40 references.

References · 20

  1. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,302
  2. 2023Tree of Thoughts: Deliberate Problem Solving with Large Language Modelscited by 3,155
  3. 2022Constitutional AI: Harmlessness from AI Feedbackcited by 2,950
  4. 2024Scaling LLM Test-Time Compute Optimally can be More Effective than Scaling Model Parameterscited by 1,928
  5. 2024The AI Scientist: Towards Fully Automated Open-Ended Scientific Discoverycited by 1,002
  6. 2024Training Large Language Models to Reason in a Continuous Latent Spacecited by 763
  7. 2019Leveraging Procedural Generation to Benchmark Reinforcement Learningcited by 478
  8. 2015Deep Recurrent Q-Learning for Partially Observable MDPscited by 457
  9. 2019Analysing Mathematical Reasoning Abilities of Neural Modelscited by 331
  10. 2017StarCraft II: A New Challenge for Reinforcement Learningcited by 268
  11. 2018BabyAI: A Platform to Study the Sample Efficiency of Grounded Language Learningcited by 238
  12. 2020The NetHack Learning Environmentcited by 196
8 more
  1. 2025Hierarchical Reasoning Modelcited by 167
  2. 2019On the Measure of Intelligencecited by 129
  3. 2021MiniHack the Planet: A Sandbox for Open-Ended Reinforcement Learning Researchcited by 96
  4. 2025Less is More: Recursive Reasoning with Tiny Networkscited by 38
  5. 2017Hindsight policy gradientscited by 36
  6. 2024Gymnasium: A Standard Interface for Reinforcement Learning Environmentscited by 35
  7. 2024Streaming Deep Reinforcement Learning Finally Workscited by 24
  8. 2024Thinking Tokens for Language Modelingcited by 23

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2025Learning When to Plan: Efficiently Allocating Test-Time Compute for LLM Agents
  2. 2026Dissecting Hierarchical Reasoning Models: A Mechanistic Study
  3. 2025Agent-as-Tool: A Study on the Hierarchical Decision Making with Reinforcement Learning
  4. 2023LgTS: Dynamic Task Sampling using LLM-generated sub-goals for Reinforcement Learning Agents
  5. 2025Emergent Hierarchical Reasoning in LLMs through Reinforcement Learning
  6. 2025RLVE: Scaling Up Reinforcement Learning for Language Models with Adaptive Verifiable Environments
  7. 2026Hierarchical Reinforcement Learning with Augmented Step-Level Transitions for LLM Agents
  8. 2025From Experience to Strategy: Empowering LLM Agents with Trainable Graph Memory