mlsanityData to 7 Oct 2026

Hierarchical Reinforcement Learning with Augmented Step-Level Transitions for LLM Agents

arXiv 2604.05808 ↗First posted 7 Apr 2026cs.AI cs.LGcited by 1

On the map: LLM agents and tool use → Reinforcement learning for LLM agents

Abstract (arXiv)

Large language model (LLM) agents have demonstrated strong capabilities in complex interactive decision-making tasks. However, existing LLM agents typically rely on increasingly long interaction histories, resulting in high computational cost and limited scalability. In this paper, we propose STEP-HRL, a hierarchical reinforcement learning (HRL) framework that enables step-level learning by conditioning only on single-step transitions rather than full interaction histories. STEP-HRL structures tasks hierarchically, using completed subtasks to represent global progress of overall task. By introducing a local progress module, it also iteratively and selectively summarizes interaction history within each subtask to produce a compact summary of local progress. Together, these components yield augmented step-level transitions for both high-level and low-level policies. Experimental results on ScienceWorld and ALFWorld benchmarks consistently demonstrate that STEP-HRL substantially outperforms baselines in terms of performance and generalization while reducing token usage. Our code is available at https://github.com/TonyStark042/STEP-HRL.

Key facts. Cited by 1 paper in the map (data through 7 Oct 2026). Placed in Reinforcement learning for LLM agents, one of 16 directions in LLM agents and tool use. Builds on 37 of its 57 references that are in the map. First posted on arXiv 7 Apr 2026.

Builds on

37 of 57 references.

Closest ancestors · 6

Same direction, newest first

  1. 2025Agentic Reinforcement Learning with Implicit Step Rewards23 Sep
  2. 2025Group-in-Group Policy Optimization for LLM Agent Training16 May
  3. 2024Enhancing Decision-Making for LLM Agents via Step-Level Q-Value Models14 Sep
  4. 2024EPO: Hierarchical LLM Agents with Environment Preference Optimization28 Aug
  5. 2024AGILE: A Novel Reinforcement Learning Framework of LLM Agents23 May
  6. 2023Language Agents with Reinforcement Learning for Strategic Play in the Werewolf Game29 Oct

Other references · 31

  1. 2017Attention Is All You Needcited by 59,932
  2. 2022Training language models to follow instructions with human feedbackcited by 15,712
  3. 2017Proximal Policy Optimization Algorithmscited by 14,441
  4. 2022ReAct: Synergizing Reasoning and Acting in Language Modelscited by 8,355
  5. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 7,916
  6. 2018Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actorcited by 4,635
  7. 2023Toolformer: Language Models Can Teach Themselves to Use Toolscited by 4,072
  8. 2023A Survey on Large Language Model based Autonomous Agentscited by 2,203
  9. 2023ToolLLM: Facilitating Large Language Models to Master 16000+ Real-world APIscited by 1,880
  10. 2021Decision Transformer: Reinforcement Learning via Sequence Modelingcited by 1,528
  11. 2021Offline Reinforcement Learning with Implicit Q-Learningcited by 1,245
  12. 2020ALFWorld: Aligning Text and Embodied Environments for Interactive Learningcited by 1,143
19 more
  1. 2025A-MEM: Agentic Memory for LLM Agentscited by 1,000
  2. 2019Advantage-Weighted Regression: Simple and Scalable Off-Policy Reinforcement Learningcited by 750
  3. 2020AWAC: Accelerating Online Reinforcement Learning with Offline Datasetscited by 647
  4. 2021Offline Reinforcement Learning as One Big Sequence Modeling Problemcited by 636
  5. 2024A Survey on the Memory Mechanism of Large Language Model based Agentscited by 622
  6. 2023FireAct: Toward Language Agent Fine-tuningcited by 247
  7. 2025MEM1: Learning to Synergize Memory and Reasoning for Efficient Long-Horizon Agentscited by 226
  8. 2022Pre-Trained Language Models for Interactive Decision-Makingcited by 220
  9. 2023SwiftSage: A Generative Agent with Fast and Slow Thinking for Complex Interactive Taskscited by 166
  10. 2022Offline RL for Natural Language Generation with Implicit Language Q Learningcited by 137
  11. 2023Large Language Models as Generalizable Policies for Embodied Taskscited by 86
  12. 2024Watch Every Step! LLM Agent Learning via Iterative Step-Level Process Refinementcited by 86
  13. 2024AvaTaR: Optimizing LLM Agents for Tool Usage via Contrastive Reasoningcited by 69
  14. 2024Agent Planning with World Knowledge Modelcited by 66
  15. 2023When Do Transformers Shine in RL? Decoupling Memory from Credit Assignmentcited by 62
  16. 2024VLM Agents Generate Their Own Memories: Distilling Experience into Embodied Programs of Thoughtcited by 55
  17. 2025Adaptive Tool Use in Large Language Models with Meta-Cognition Triggercited by 29
  18. 2024ACC-Collab: An Actor-Critic Approach to Multi-Agent LLM Collaborationcited by 22
  19. 2024Efficient Recurrent Off-Policy RL Requires a Context-Encoder-Specific Learning Ratecited by 10

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2026: 1.

1 paper.

  1. 2026HIPIF: Hierarchical Planning and Information Folding for Long-Horizon LLM Agent Learningcited by 1

Similar, not cited

Close in wording, no citation either way

  1. 2026Hierarchical Control in Multi-Agent Games: LLM-based Planning and RL Execution
  2. 2024From Novice to Expert: LLM Agent Policy Optimization via Step-wise Reinforcement Learning
  3. 2026Hybrid LLM-Augmented Reinforcement Learning Agents for Complex Sequential Decision Tasks
  4. 2026HiMAC: Hierarchical Macro-Micro Learning for Long-Horizon LLM Agents
  5. 2026Hierarchical Reinforcement Learning with Stable Temporal Abstraction for Language Model Agents
  6. 2024Exploring the limits of Hierarchical World Models in Reinforcement Learning
  7. 2024StepTool: Enhancing Multi-Step Tool Usage in LLMs via Step-Grained Reinforcement Learning
  8. 2026T2SPO: Trajectory-to-Step Policy Optimization for Agentic Reinforcement Learning