HISPO: Hierarchical Importance-Sampling Policy Optimization with Entropy-Derived Segments
On the map: LLM reasoning → Reinforcement learning for reasoning
Abstract (arXiv)
Reinforcement learning with verifiable rewards (RLVR) has become a central approach for improving mathematical reasoning in language models, but long-form completions introduce a difficult credit-assignment problem: different parts of a solution trace may contribute unevenly to final correctness. Existing policyoptimization objectives for RLVR commonly apply importance-sampling correction at either the token level (GRPO, DAPO) or the sequence level (GSPO), imposing different granularities for assigning credit across a response. We introduce Hierarchical Importance-Sampling Policy Optimization (HISPO), a segment-level policy-optimization method that constructs rollout-time entropy-derived contiguous segments, assigns soft entropy-based saliency weights, and applies clipped importance-sampling correction at the segment granularity. This provides an intermediate correction unit between token-level GRPO/DAPO and sequence-level GSPO. We evaluate HISPO by fine-tuning Qwen3-1.7B-Base on mathematical reasoning tasks. Across six benchmarks, HISPO improves Pass@8 over the strongest baseline on all benchmarks and matches or exceeds the strongest baseline in Acc@8 on five of them. On AIME25, HISPO improves over GRPO by +3.75 Acc@8 and +3.78 Pass@8, and over GSPO by +2.50 Acc@8 and +1.27 Pass@8. These results suggest that segment-level correction is a promising granularity for RLVR in long-form mathematical reasoning.
Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in Reinforcement learning for reasoning, one of 17 directions in LLM reasoning. Builds on 19 of its 28 references that are in the map. First posted on arXiv 14 Sep 2026.
Builds on
19 of 28 references.
Closest ancestors · 12
Same direction, newest first
- 2026Length-Unbiased Sequence Policy Optimization: Revealing and Controlling Response Length Variation in RLVR5 Feb
- 2026Orchestrating Tokens and Sequences: Dynamic Hybrid Policy Optimization for RLVR9 Jan
- 2025No Prompt Left Behind: Exploiting Zero-Variance Prompts in LLM Reinforcement Learning via Entropy-Guided Advantage Shaping26 Sep
- 2025Reasoning with Exploration: An Entropy Perspective17 Jun
- 2025Beyond the 80/20 Rule: High-Entropy Minority Tokens Drive Effective Reinforcement Learning for LLM Reasoning2 Jun
- 2025Segment Policy Optimization: Effective Segment-Level Credit Assignment in RL for Large Language Models29 May
- 2025The Entropy Mechanism of Reinforcement Learning for Reasoning Language Models28 May
- 2025DAPO: An Open-Source LLM Reinforcement Learning System at Scale18 Mar
- 2025DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning22 Jan
- 2023Math-Shepherd: Verify and Reinforce LLMs Step-by-step without Human Annotations14 Dec
- 2023Let's Verify Step by Step31 May
- 2017Proximal Policy Optimization Algorithms20 Jul
Other references · 7
- 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 9,379
- 2025Qwen3 Technical Reportcited by 8,820
- 2021Measuring Mathematical Problem Solving With the MATH Datasetcited by 5,371
- 2024HybridFlow: A Flexible and Efficient RLHF Frameworkcited by 2,057
- 2022Solving Quantitative Reasoning Problems with Language Modelscited by 1,392
- 2024OlympiadBench: A Challenging Benchmark for Promoting AGI with Olympiad-Level Bilingual Multimodal Scientific Problemscited by 1,136
- 2025Group Sequence Policy Optimizationcited by 835
Built on by
Not cited yet.
Similar, not cited
Close in wording, no citation either way
- 2025ESPO: Entropy Importance Sampling Policy Optimization
- 2026Learning from Failures: Correction-Oriented Policy Optimization with Verifiable Rewards
- 2025SSPO: Subsentence-level Policy Optimization
- 2026Revisiting Reinforcement Learning with Verifiable Rewards from a Contrastive Perspective
- 2026MCPO: Mastery-Consolidated Policy Optimization for Large Reasoning Models
- 2026Unifying Group-Relative and Self-Distillation Policy Optimization via Sample Routing
- 2025Slow-Fast Policy Optimization: Reposition-Before-Update for LLM Reasoning
- 2025GTPO and GRPO-S: Token and Sequence-Level Reward Shaping with Policy Entropy