mlsanityData to 13 Oct 2026

Hierarchical Reinforcement Learning with Stable Temporal Abstraction for Language Model Agents

arXiv 2610.05473 ↗First posted 4 Oct 2026cs.AIcited by 0

On the map: LLM agents and tool use → Reinforcement learning for LLM agents

Abstract (arXiv)

Hierarchical reinforcement learning improves long-horizon control by organizing primitive actions around persistent subgoals and assigning credit at multiple temporal scales. Recent hierarchical language agents bring these benefits to interactive tasks by explicitly separating subgoal planning from action execution. We observe, however, that an explicit hierarchy does not by itself determine how stable the resulting temporal abstraction is: the learned boundary policy may replace the subgoal almost every turn, making it effectively transient, or retain a subgoal after it has stopped being appropriate. We call this temporal abstraction instability. We propose Stable Temporal Abstraction via Constrained Optimization (STAC), a constrained boundary-policy optimization method that represents premature replanning and stale persistence as constraint costs. STAC applies the resulting Lagrangian costs only to the sampled boundary decision, leaving the underlying algorithm's rewards, critic targets, subgoal advantages, and primitive-action advantages unchanged. Across two backbones and two benchmarks, STAC improves success over a strong hierarchical baseline by 8.1 and 7.9 points on ALFWorld and WebShop with Qwen3-0.6B, and by 23.5 and 15.8 points with Llama-3.2-1B-Instruct.

Key facts. No citing papers in the map yet (data through 13 Oct 2026). Placed in Reinforcement learning for LLM agents, one of 16 directions in LLM agents and tool use. Builds on 19 of its 25 references that are in the map. First posted on arXiv 4 Oct 2026.

Builds on

19 of 25 references.

Closest ancestors · 6

Same direction, newest first

  1. 2026HIPIF: Hierarchical Planning and Information Folding for Long-Horizon LLM Agent Learning9 Jun
  2. 2026Multi2: Hierarchical Multi-Agent Decision-Making with LLM-Based Agents in Interactive Environments2 Jun
  3. 2026Hierarchical Reinforcement Learning with Augmented Step-Level Transitions for LLM Agents7 Apr
  4. 2026HiMAC: Hierarchical Macro-Micro Learning for Long-Horizon LLM Agents1 Mar
  5. 2025Group-in-Group Policy Optimization for LLM Agent Training16 May
  6. 2024ArCHer: Training Language Model Agents via Hierarchical Multi-Turn RL29 Feb

Other references · 13

  1. 2017Proximal Policy Optimization Algorithmscited by 14,592
  2. 2024The Llama 3 Herd of Modelscited by 14,486
  3. 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 9,091
  4. 2015High-Dimensional Continuous Control Using Generalized Advantage Estimationcited by 1,940
  5. 2022WebShop: Towards Scalable Real-World Web Interaction with Grounded Language Agentscited by 1,219
  6. 2020ALFWorld: Aligning Text and Embodied Environments for Interactive Learningcited by 1,167
  7. 2018Data-Efficient Hierarchical Reinforcement Learningcited by 442
  8. 2016The Option-Critic Architecturecited by 439
  9. 2017When Waiting is not an Option : Learning Options with a Deliberation Costcited by 58
  10. 2020Generating Adjacency-Constrained Subgoals in Hierarchical Reinforcement Learningcited by 57
  11. 2019Hierarchical Reinforcement Learning with Advantage-Based Auxiliary Rewardscited by 39
  12. 2021Hierarchical Reinforcement Learning with Timed Subgoalscited by 21
1 more
  1. 2022DHRL: A Graph-Based Approach for Long-Horizon and Sparse Hierarchical Reinforcement Learningcited by 19

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2024Exploring the limits of Hierarchical World Models in Reinforcement Learning
  2. 2025Stabilizing Reinforcement Learning with LLMs: Formulation and Practices
  3. 2018Hierarchical Imitation and Reinforcement Learning
  4. 2026Abstraction for Offline Goal-Conditioned Reinforcement Learning
  5. 2026HiPER: Hierarchical Reinforcement Learning with Explicit Credit Assignment for Large Language Model Agents
  6. 2025CoDA: A Context-Decoupled Hierarchical Agent with Reinforcement Learning
  7. 2026Adaptive Coarse-to-Fine Subgoal Refinement for Long-Horizon Offline Goal-Conditioned Reinforcement Learning
  8. 2025CPGD: Toward Stable Rule-based Reinforcement Learning for Language Models