mlsanityData to 14 Oct 2026

STARE: Surprisal-Guided Token-Level Advantage Reweighting for Policy Entropy Stability

arXiv 2606.19236 ↗First posted 17 Jun 2026cs.LG cs.AI cs.CLcited by 3

On the map: LLM reasoning → Reinforcement learning for reasoning

Abstract (arXiv)

Reinforcement Learning with Verifiable Rewards algorithms like GRPO have emerged as the dominant post-training paradigm for complex reasoning in LLMs, yet commonly suffer from policy entropy collapse during training. We conduct a first-order gradient analysis of token-level entropy dynamics under GRPO and identify a token-level credit assignment mismatch: the per-token entropy variation decomposes into the product of the trajectory-level advantage and an entropy sensitivity function over the next-token distribution, yielding an advantage-surprisal four-quadrant structure and a near-criticality property. Motivated by it, we propose STARE (Surprisal-guided Token-level Advantage Reweighting for policy Entropy stability), which identifies entropy-critical token subsets via batch-internal surprisal quantiles, selectively reweights their effective advantages, and incorporates a target-entropy closed-loop gate for stable entropy regulation. Across model scales from 1.5B to 32B and three task families (Short CoT, Long CoT, and Multi-Turn Tool Use), STARE sustains stable RL training over thousands of steps while maintaining policy entropy within the target band. On AIME24 and AIME25, STARE outperforms DAPO and other competitive baselines by 4%-8% in average accuracy, with reflection tokens and response length growing in tandem, indicating sustained exploration-exploitation balance that further unlocks RL training potential.Code is available at https://github.com/hp-luo/STARE.

Key facts. Cited by 3 papers in the map (data through 14 Oct 2026). Placed in Reinforcement learning for reasoning, one of 17 directions in LLM reasoning. Builds on 52 of its 65 references that are in the map. First posted on arXiv 17 Jun 2026.

Builds on

52 of 65 references.

Closest ancestors · 19

Same direction, newest first

  1. 2026From log π to π: Taming Divergence in Soft Clipping via Bilateral Decoupled Decay of Probability Gradient Weight15 Mar
  2. 2026STAPO: Stabilizing Reinforcement Learning for LLMs by Silencing Rare Spurious Tokens17 Feb
  3. 2026Flexible Entropy Control in RLVR with a Gradient-Preserving Perspective10 Feb
  4. 2025Rethinking Sample Polarity in Reinforcement Learning with Verifiable Rewards25 Dec
  5. 2025Revisiting Entropy in Reinforcement Learning for Large Reasoning Models8 Nov
  6. 2025Rethinking Entropy Interventions in RLVR: An Entropy Change Perspective11 Oct
  7. 2025Low-probability Tokens Sustain Exploration in Reinforcement Learning with Verifiable Reward3 Oct
  8. 2025CE-GPPO: Coordinating Entropy via Gradient-Preserving Clipping Policy Optimization in Reinforcement Learning25 Sep
  9. 2025SimpleTIR: End-to-End Reinforcement Learning for Multi-Turn Tool-Integrated Reasoning2 Sep
  10. 2025Reasoning with Exploration: An Entropy Perspective17 Jun
  11. 2025Rewarding the Unlikely: Lifting GRPO Beyond Distribution Sharpening3 Jun
  12. 2025Beyond the 80/20 Rule: High-Entropy Minority Tokens Drive Effective Reinforcement Learning for LLM Reasoning2 Jun
7 more
  1. 2025The Surprising Effectiveness of Negative Reinforcement in LLM Reasoning2 Jun
  2. 2025The Entropy Mechanism of Reinforcement Learning for Reasoning Language Models28 May
  3. 2025On the Effect of Negative Gradient in Group Relative Deep Reinforcement Optimization24 May
  4. 2025Do Not Let Low-Probability Tokens Over-Dominate in RL for LLMs19 May
  5. 2025Does Reinforcement Learning Really Incentivize Reasoning Capacity in LLMs Beyond the Base Model?18 Apr
  6. 2025VAPO: Efficient and Reliable Reinforcement Learning for Advanced Reasoning Tasks7 Apr
  7. 2017Proximal Policy Optimization Algorithms20 Jul

Other references · 33

  1. 2023GPT-4 Technical Reportcited by 18,396
  2. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,459
  3. 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 9,379
  4. 2021Measuring Mathematical Problem Solving With the MATH Datasetcited by 5,371
  5. 2018Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actorcited by 4,885
  6. 2022Solving Quantitative Reasoning Problems with Language Modelscited by 1,392
  7. 2024OlympiadBench: A Challenging Benchmark for Promoting AGI with Olympiad-Level Bilingual Multimodal Scientific Problemscited by 1,136
  8. 2023WizardLM: Empowering large pre-trained language models to follow complex instructionscited by 1,125
  9. 2022Scaling Laws for Reward Model Overoptimizationcited by 1,095
  10. 2024Qwen2.5-Math Technical Report: Toward Mathematical Expert Model via Self-Improvementcited by 1,090
  11. 2025Group Sequence Policy Optimizationcited by 835
  12. 2023WizardCoder: Empowering Code Large Language Models with Evol-Instructcited by 733
21 more
  1. 2023WizardMath: Empowering Mathematical Reasoning for Large Language Models via Reinforced Evol-Instructcited by 625
  2. 2025ReTool: Reinforcement Learning for Strategic Tool Use in LLMscited by 392
  3. 2025Demystifying Long Chain-of-Thought Reasoning in LLMscited by 326
  4. 2025MiniMax-M1: Scaling Test-Time Compute Efficiently with Lightning Attentioncited by 263
  5. 2024VinePPO: Refining Credit Assignment in RL Training of LLMscited by 183
  6. 2025AIMO-2 Winning Solution: Building State-of-the-Art Mathematical Reasoning Models with OpenMathReasoning datasetcited by 174
  7. 2025Skywork Open Reasoner 1 Technical Reportcited by 165
  8. 2025JustRL: Scaling a 1.5B LLM with a Simple RL Recipecited by 64
  9. 2025Agent RL Scaling Law: Agent RL with Spontaneous Code Execution for Mathematical Problem Solvingcited by 62
  10. 2025BAPO: Stabilizing Off-Policy Reinforcement Learning for LLMs via Balanced Policy Optimization with Adaptive Clippingcited by 47
  11. 2023Transformer-Based Language Model Surprisal Predicts Human Reading Times Best with About Two Billion Training Tokenscited by 41
  12. 2024Frequency Explains the Inverse Correlation of Large Language Models' Size, Training Data Amount, and Surprisal's Fit to Reading Timescited by 39
  13. 2025Pruning the Unsurprising: Efficient LLM Reasoning via First-Token Surprisalcited by 32
  14. 2025Hunyuan-TurboS: Advancing Large Language Models through Mamba-Transformer Synergy and Adaptive Chain-of-Thoughtcited by 29
  15. 2024Arena Learning: Build Data Flywheel for LLMs Post-training via Simulated Chatbot Arenacited by 26
  16. 2025Towards Effective Code-Integrated Reasoningcited by 23
  17. 2025EPO: Entropy-regularized Policy Optimization for LLM Agents Reinforcement Learningcited by 17
  18. 2025AgentMath: Empowering Mathematical Reasoning for Large Language Models via Tool-Augmented Agentcited by 17
  19. 2025EntroPIC: Towards Stable Long-Term Training of LLMs via Entropy Stabilization with Proportional-Integral Controlcited by 13
  20. 2026OffSeeker: Online Reinforcement Learning Is Not All You Need for Deep Research Agentscited by 12
  21. 2026On the Entropy Dynamics in Reinforcement Fine-Tuning of Large Language Modelscited by 8

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 3.

3 papers.

  1. 2026Granularity-Adaptive Credit Assignment for Long-Horizon LLM Agent Reinforcement Learningcited by 2
  2. 2026Video-HopChain: Multi-Hop Questions and Confidence-Gated Exploration for Video Reasoning Modelscited by 0
  3. 2026GRPODropout: Less is More for Online Reinforcement Learning Rolloutscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026ERPO: Token-Level Entropy-Regulated Policy Optimization for Large Reasoning Models
  2. 2025Beyond High-Entropy Exploration: Correctness-Aware Low-Entropy Segment-Based Advantage Shaping for Reasoning LLMs
  3. 2026Surprising Success, Repeated Failure: Entropy-Guided Credit Assignment for Exploration in LLM Reasoning
  4. 2025GTPO and GRPO-S: Token and Sequence-Level Reward Shaping with Policy Entropy
  5. 2025Heterogeneous Adaptive Policy Optimization: Tailoring Optimization to Every Token's Nature
  6. 2025Efficient Reinforcement Learning with Semantic and Token Entropy for LLM Reasoning
  7. 2025Quantile Advantage Estimation: Stabilizing RLVR for LLM Reasoning
  8. 2026EP-GRPO: Entropy-Progress Aligned Group Relative Policy Optimization with Implicit Process Guidance