mlsanityData to 14 Oct 2026

Emergent Hierarchical Reasoning in LLMs through Reinforcement Learning

arXiv 2509.03646 ↗First posted 3 Sep 2025cs.AI cs.CLcited by 42

On the map: LLM reasoning → Reinforcement learning for reasoning

Abstract (arXiv)

Reinforcement Learning (RL) has proven highly effective at enhancing the complex reasoning abilities of Large Language Models (LLMs), yet underlying mechanisms driving this success remain largely opaque. Our analysis reveals that puzzling phenomena like ``aha moments", ``length-scaling'' and entropy dynamics are not disparate occurrences but hallmarks of an emergent reasoning hierarchy, akin to the separation of high-level strategic planning from low-level procedural execution in human cognition. We uncover a compelling two-phase dynamic: initially, a model is constrained by procedural correctness and must improve its low-level skills. The learning bottleneck then decisively shifts, with performance gains being driven by the exploration and mastery of high-level strategic planning. This insight exposes a core inefficiency in prevailing RL algorithms like GRPO, which apply optimization pressure agnostically and dilute the learning signal across all tokens. To address this, we propose Hierarchy-Aware Credit Assignment (HICRA), an algorithm that concentrates optimization efforts on high-impact planning tokens. Our extensive experiments validate that HICRA significantly outperforms strong baselines, and offer deep insights into how reasoning advances through the lens of strategic exploration.

Key facts. Cited by 42 papers in the map (data through 14 Oct 2026). Placed in Reinforcement learning for reasoning, one of 17 directions in LLM reasoning. Builds on 30 of its 44 references that are in the map. First posted on arXiv 3 Sep 2025.

Builds on

30 of 44 references.

Closest ancestors · 10

Same direction, newest first

  1. 2025Reasoning with Exploration: An Entropy Perspective17 Jun
  2. 2025Beyond the 80/20 Rule: High-Entropy Minority Tokens Drive Effective Reinforcement Learning for LLM Reasoning2 Jun
  3. 2025The Entropy Mechanism of Reinforcement Learning for Reasoning Language Models28 May
  4. 2025SEED-GRPO: Semantic Entropy Enhanced GRPO for Uncertainty-Aware Policy Optimization18 May
  5. 2025Understanding R1-Zero-Like Training: A Critical Perspective26 Mar
  6. 2025DAPO: An Open-Source LLM Reinforcement Learning System at Scale18 Mar
  7. 2025DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning22 Jan
  8. 2025Kimi k1.5: Scaling Reinforcement Learning with LLMs22 Jan
  9. 2023Let's Verify Step by Step31 May
  10. 2017Proximal Policy Optimization Algorithms20 Jul

Other references · 20

  1. 2024The Llama 3 Herd of Modelscited by 14,715
  2. 2025Qwen3 Technical Reportcited by 8,820
  3. 2021Evaluating Large Language Models Trained on Codecited by 8,320
  4. 2025Qwen2.5-VL Technical Reportcited by 5,587
  5. 2024Qwen2.5 Technical Reportcited by 4,942
  6. 2023MathVista: Evaluating Mathematical Reasoning of Foundation Models in Visual Contextscited by 1,614
  7. 2024OpenAI o1 System Cardcited by 1,244
  8. 2024OlympiadBench: A Challenging Benchmark for Promoting AGI with Olympiad-Level Bilingual Multimodal Scientific Problemscited by 1,136
  9. 2018Reinforcement Learning and Control as Probabilistic Inference: Tutorial and Reviewcited by 482
  10. 2025VL-Rethinker: Incentivizing Self-Reflection of Vision-Language Models with Reinforcement Learningcited by 323
  11. 2025Pixel Reasoner: Incentivizing Pixel-Space Reasoning with Curiosity-Driven Reinforcement Learningcited by 278
  12. 2025Hierarchical Reasoning Modelcited by 177
8 more
  1. 2025Can MLLMs Reason in Multimodality? EMMA: An Enhanced MultiModal ReAsoning Benchmarkcited by 134
  2. 2025MiMo-VL Technical Reportcited by 115
  3. 2025Beyond Distillation: Pushing the Limits of Medical LLM Reasoning with Minimalist Rule-Based RLcited by 36
  4. 2025To Code or not to Code? Adaptive Tool Integration for Math Language Models via Expectation-Maximizationcited by 25
  5. 2025Learning to Reason via Mixture-of-Thought for Logical Reasoningcited by 20
  6. 2025HS-STaR: Hierarchical Sampling for Self-Taught Reasoners via Difficulty Estimation and Budget Reallocationcited by 18
  7. 2025R1-RE: Cross-Domain Relation Extraction with RLVRcited by 11
  8. 2025Afford-X: Generalizable and Slim Affordance Reasoning for Task-oriented Manipulationcited by 3

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 18; 2026: 24.

42 papers.

Sort:

By area:

  1. 2025ReasoningBank: Scaling Agent Self-Evolving with Reasoning Memorycited by 284
  2. 2025A Survey of Reinforcement Learning for Large Reasoning Modelscited by 180
  3. 2025Agent0: Unleashing Self-Evolving Agents from Zero Data via Tool-Integrated Reasoningcited by 75
  4. 2025Rethinking Entropy Interventions in RLVR: An Entropy Change Perspectivecited by 52
  5. 2026Embarrassingly Simple Self-Distillation Improves Code Generationcited by 40
  6. 2025NewtonBench: Benchmarking Generalizable Scientific Law Discovery in LLM Agentscited by 36
  7. 2025RL Squeezes, SFT Expands: A Comparative Study of Reasoning LLMscited by 34
  8. 2025Demystifying Reinforcement Learning in Agentic Reasoningcited by 33
  9. 2025Reverse-Engineered Reasoning for Open-Ended Generationcited by 32
  10. 2026From Reasoning to Agentic: Credit Assignment in Reinforcement Learning for Large Language Modelscited by 23
  11. 2025The Debate on RLVR Reasoning Capability Boundary: Shrinkage, Expansion, or Both? A Two-Stage Dynamic Viewcited by 13
  12. 2026On the Direction of RLVR Updates for LLM Reasoning: Identification and Exploitationcited by 13
30 more
  1. 2025From Illusion to Intention: Visual Rationale Learning for Vision-Language Reasoningcited by 9
  2. 2026Step Potential Advantage Estimation: Harnessing Intermediate Confidence and Correctness for Efficient Mathematical Reasoningcited by 9
  3. 2026From Narrow to Panoramic Vision: Attention-Guided Cold-Start Reshapes Multimodal Reasoningcited by 8
  4. 2026Search Beyond What Can Be Taught: Evolving the Knowledge Boundary in Agentic Visual Generationcited by 8
  5. 2026Bad Seeing or Bad Thinking? Rewarding Perception for Multimodal Reasoningcited by 7
  6. 2025Training Large Language Models To Reason In Parallel With Global Forking Tokenscited by 6
  7. 2026Towards a Mechanistic Understanding of Large Reasoning Models: A Survey of Training, Inference, and Failurescited by 6
  8. 2025ReLaX: Reasoning with Latent Exploration for Large Reasoning Modelscited by 5
  9. 2026Beyond Uniform Credit Assignment: Selective Eligibility Traces for RLVRcited by 5
  10. 2026ReasoningFlow: Discourse Structures for Understanding LLM Reasoning Tracescited by 5
  11. 2025CogDoc: Towards Unified thinking in Documentscited by 3
  12. 2026G2TR: Generation-Guided Visual Token Reduction for Separate-Encoder Unified Multimodal Modelscited by 3
  13. 2025Explore-Execute Chain: Towards an Efficient Structured Reasoning Paradigmcited by 2
  14. 2025Selection, Reflection and Self-Refinement: Revisit Reasoning Tasks via a Causal Lenscited by 2
  15. 2025The Fragility of Strategic Thinking in Large Language Modelscited by 2
  16. 2026Measuring and Mitigating Post-hoc Rationalization in Reverse Chain-of-Thought Generationcited by 2
  17. 2026RL Post-Training Builds Compositional Reasoning Strategiescited by 2
  18. 2025Language-Driven Object-Oriented Two-Stage Method for Scene Graph Anticipationcited by 1
  19. 2025Does RLVR Extend Reasoning Boundaries? Investigating Capability Expansion in Vision-Language Modelscited by 1
  20. 2026Dynamical Systems Theory Behind a Hierarchical Reasoning Modelcited by 1
  21. 2026Hierarchical Chain-of-Thought: Enhancing LLM Reasoning Performance and Efficiencycited by 1
  22. 2026Asymmetric Advantage Modulation Calibrates Entropy Dynamics in RLVRcited by 1
  23. 2026Skip-Connected Policy Optimization for Implicit Advantagecited by 1
  24. 2026Fine-Tuning Small Reasoning Models for Quantum Field Theorycited by 1
  25. 2026General Preference Reinforcement Learningcited by 1
  26. 2026GMTS: Gradient Magnitude-based Token Selection Improves RLVR Training for LLM Reasoningcited by 1
  27. 2026Starve to Perceive: Taming Lazy Perception in VLMs with Constrained Visual Bandwidthcited by 0
  28. 2026Interpreting FCDNNs via RG on Exponential Familycited by 0
  29. 2026Cognitive Convergence: Deep Similarities Between Large Language Models and Human Cognitioncited by 0
  30. 2026Dependency-Aware Reward Shaping for Agentic Reinforcement Learningcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Hierarchical Budget Policy Optimization for Adaptive Reasoning
  2. 2025Agent-as-Tool: A Study on the Hierarchical Decision Making with Reinforcement Learning
  3. 2025Scaling Up RL: Unlocking Diverse Reasoning in LLMs via Prolonged Training
  4. 2025Divide and Conquer: Grounding LLMs as Efficient Decision-Making Agents via Offline Hierarchical Reinforcement Learning
  5. 2025CoDA: A Context-Decoupled Hierarchical Agent with Reinforcement Learning
  6. 2025Rectifying LLM Thought from Lens of Optimization
  7. 2026HiPER: Hierarchical Reinforcement Learning with Explicit Credit Assignment for Large Language Model Agents
  8. 2025How Difficulty-Aware Staged Reinforcement Learning Enhances LLMs' Reasoning Capabilities: A Preliminary Experimental Study