# Implicit Hierarchical GRPO: Decoupling Tool Invocation from Execution for Tool-Integrated Mathematical Reasoning

arXiv: https://arxiv.org/abs/2605.18500 · HTML: https://mlsanity.com/paper/2605.18500/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Placed in Reasoning for agents, planning and code, one of 17 directions in LLM reasoning.
- Builds on 40 of its 54 references that are in the map.
- First posted on arXiv 18 May 2026.

## Placed in

- LLM reasoning / [Reasoning for agents, planning and code](https://mlsanity.com/lr/reasoning-for-agents-planning-and-code/)

## Builds on (most cited first, 30 of 40)

- [DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models](https://mlsanity.com/paper/2402.03300/) (2024)
- [Qwen3 Technical Report](https://mlsanity.com/paper/2505.09388/) (2025)
- [Measuring Mathematical Problem Solving With the MATH Dataset](https://mlsanity.com/paper/2103.03874/) (2021)
- [DAPO: An Open-Source LLM Reinforcement Learning System at Scale](https://mlsanity.com/paper/2503.14476/) (2025)
- [HybridFlow: A Flexible and Efficient RLHF Framework](https://mlsanity.com/paper/2409.19256/) (2024)
- [Challenging BIG-Bench Tasks and Whether Chain-of-Thought Can Solve Them](https://mlsanity.com/paper/2210.09261/) (2022)
- [Search-R1: Training LLMs to Reason and Leverage Search Engines with Reinforcement Learning](https://mlsanity.com/paper/2503.09516/) (2025)
- [Understanding R1-Zero-Like Training: A Critical Perspective](https://mlsanity.com/paper/2503.20783/) (2025)
- [SimpleRL-Zoo: Investigating and Taming Zero Reinforcement Learning for Open Base Models in the Wild](https://mlsanity.com/paper/2503.18892/) (2025)
- [ReTool: Reinforcement Learning for Strategic Tool Use in LLMs](https://mlsanity.com/paper/2504.11536/) (2025)
- [R1-Searcher: Incentivizing the Search Capability in LLMs via Reinforcement Learning](https://mlsanity.com/paper/2503.05592/) (2025)
- [ToRA: A Tool-Integrated Reasoning Agent for Mathematical Problem Solving](https://mlsanity.com/paper/2309.17452/) (2023)
- [ReWOO: Decoupling Reasoning from Observations for Efficient Augmented Language Models](https://mlsanity.com/paper/2305.18323/) (2023)
- [Hierarchical Reasoning Model](https://mlsanity.com/paper/2506.21734/) (2025)
- [Agentic Reinforced Policy Optimization](https://mlsanity.com/paper/2507.19849/) (2025)
- [ToRL: Scaling Tool-Integrated RL](https://mlsanity.com/paper/2503.23383/) (2025)
- [SimpleTIR: End-to-End Reinforcement Learning for Multi-Turn Tool-Integrated Reasoning](https://mlsanity.com/paper/2509.02479/) (2025)
- [Chain of Code: Reasoning with a Language Model-Augmented Code Emulator](https://mlsanity.com/paper/2312.04474/) (2023)
- [Agent Lightning: Train ANY AI Agents with Reinforcement Learning](https://mlsanity.com/paper/2508.03680/) (2025)
- [ReMA: Learning to Meta-think for LLMs with Multi-Agent Reinforcement Learning](https://mlsanity.com/paper/2503.09501/) (2025)
- [LogiQA: A Challenge Dataset for Machine Reading Comprehension with Logical Reasoning](https://mlsanity.com/paper/2007.08124/) (2020)
- [OctoTools: An Agentic Framework with Extensible Tools for Complex Reasoning](https://mlsanity.com/paper/2502.11271/) (2025)
- [Why Does Hierarchy (Sometimes) Work So Well in Reinforcement Learning?](https://mlsanity.com/paper/1909.10618/) (2019)
- [Beyond Examples: High-level Automated Reasoning Paradigm in In-Context Learning via MCTS](https://mlsanity.com/paper/2411.18478/) (2024)
- [rStar2-Agent: Agentic Reasoning Technical Report](https://mlsanity.com/paper/2508.20722/) (2025)
- [Understanding Tool-Integrated Reasoning](https://mlsanity.com/paper/2508.19201/) (2025)
- [Your Group-Relative Advantage Is Biased](https://mlsanity.com/paper/2601.08521/) (2026)
- [TemplateRL: Structured Template-Guided Reinforcement Learning for LLM Reasoning](https://mlsanity.com/paper/2505.15692/) (2025)
- [Towards Effective Code-Integrated Reasoning](https://mlsanity.com/paper/2505.24480/) (2025)
- [AgentMath: Empowering Mathematical Reasoning for Large Language Models via Tool-Augmented Agent](https://mlsanity.com/paper/2512.20745/) (2025)

## Built on by (most cited first, 1 of 1)

- [When Not to Imitate: Boundary-Aware Skill Memory for Reliable Tool-Use LLM Agents](https://mlsanity.com/paper/2608.22339/) (2026)

## Abstract (arXiv)

Large language models (LLMs) have increasingly leveraged tool invocation to enhance their reasoning capabilities. However, existing approaches typically tightly couple tool invocation with immediate execution. Such immediate tool interaction may disrupt the reasoning coherence of LLMs and constrain their expressivity, ultimately degrading reasoning performance. To this end, for the first time, we propose and formalize the problem of decoupling tool invocation from execution during reasoning, and introduce delayed execution with explicit control to enhance tool-integrated reasoning (TIR). Furthermore, we propose a hierarchical control framework and theoretically derive a surrogate loss that enables an implicitly hierarchical policy to learn behavior equivalent to that of an explicit hierarchical policy, leading to the proposed IH-GRPO algorithm. Extensive experiments on IH-GRPO achieve absolute improvements of 1.87%, 2.16%, and 2.53% on Qwen3-1.7B, Qwen3-4B, and Qwen3-8B across six out-of-domain mathematical reasoning benchmarks over the strongest baseline method, while also yielding consistent performance gains in other domains. Our code is available at https://github.com/Lumina04/IH-GRPO-01.