mlsanityData to 13 Oct 2026

Beyond the 80/20 Rule: High-Entropy Minority Tokens Drive Effective Reinforcement Learning for LLM Reasoning

arXiv 2506.01939 ↗First posted 2 Jun 2025cs.CL cs.AI cs.LGcited by 548

On the map: LLM reasoning → Reinforcement learning for reasoning

Abstract (arXiv)

Reinforcement Learning with Verifiable Rewards (RLVR) has emerged as a powerful approach to enhancing the reasoning capabilities of Large Language Models (LLMs), while its mechanisms are not yet well understood. In this work, we undertake a pioneering exploration of RLVR through the novel perspective of token entropy patterns, comprehensively analyzing how different tokens influence reasoning performance. By examining token entropy patterns in Chain-of-Thought (CoT) reasoning, we observe that only a small fraction of tokens exhibit high entropy, and these tokens act as critical forks that steer the model toward diverse reasoning pathways. Furthermore, studying how entropy patterns evolve during RLVR training reveals that RLVR largely adheres to the base model's entropy patterns, primarily adjusting the entropy of high-entropy tokens. These findings highlight the significance of high-entropy tokens (i.e., forking tokens) to RLVR. We ultimately improve RLVR by restricting policy gradient updates to forking tokens and uncover a finding even beyond the 80/20 rule: utilizing only 20% of the tokens while maintaining performance comparable to full-gradient updates on the Qwen3-8B base model and significantly surpassing full-gradient updates on the Qwen3-32B (+11.04 on AIME'25 and +7.71 on AIME'24) and Qwen3-14B (+4.79 on AIME'25 and +5.21 on AIME'24) base models, highlighting a strong scaling trend. In contrast, training exclusively on the 80% lowest-entropy tokens leads to a marked decline in performance. These findings indicate that the efficacy of RLVR primarily arises from optimizing the high-entropy tokens that decide reasoning directions. Collectively, our results highlight the potential to understand RLVR through a token-entropy perspective and optimize RLVR by leveraging high-entropy minority tokens to further improve LLM reasoning.

Key facts. Cited by 548 papers in the map (data through 13 Oct 2026). Placed in Reinforcement learning for reasoning, one of 17 directions in LLM reasoning. Builds on 36 of its 47 references that are in the map. First posted on arXiv 2 Jun 2025.

Builds on

36 of 47 references.

Closest ancestors · 15

Same direction, newest first

  1. 2025DanceGRPO: Unleashing GRPO on Visual Generation12 May
  2. 2025Flow-GRPO: Training Flow Matching Models via Online RL8 May
  3. 2025Absolute Zero: Reinforced Self-play Reasoning with Zero Data6 May
  4. 2025Reinforcement Learning for Reasoning in Large Language Models with One Training Example29 Apr
  5. 2025Does Reinforcement Learning Really Incentivize Reasoning Capacity in LLMs Beyond the Base Model?18 Apr
  6. 2025VAPO: Efficient and Reliable Reinforcement Learning for Advanced Reasoning Tasks7 Apr
  7. 2025Open-Reasoner-Zero: An Open Source Approach to Scaling Up Reinforcement Learning on the Base Model31 Mar
  8. 2025SimpleRL-Zoo: Investigating and Taming Zero Reinforcement Learning for Open Base Models in the Wild24 Mar
  9. 2025DAPO: An Open-Source LLM Reinforcement Learning System at Scale18 Mar
  10. 2025Cognitive Behaviors that Enable Self-Improving Reasoners, or, Four Habits of Highly Effective STaRs3 Mar
  11. 2025Ignore the KL Penalty! Boosting Exploration on Critical Tokens to Enhance RL Fine-Tuning10 Feb
  12. 2025Kimi k1.5: Scaling Reinforcement Learning with LLMs22 Jan
3 more
  1. 2024Tulu 3: Pushing Frontiers in Open Language Model Post-Training22 Nov
  2. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Model29 May
  3. 2017Proximal Policy Optimization Algorithms20 Jul

Other references · 21

  1. 2022Training language models to follow instructions with human feedbackcited by 15,856
  2. 2015Distilling the Knowledge in a Neural Networkcited by 10,180
  3. 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 9,091
  4. 2025Qwen3 Technical Reportcited by 8,510
  5. 2023Visual Instruction Tuningcited by 7,827
  6. 2025Qwen2.5-VL Technical Reportcited by 5,499
  7. 2021Measuring Mathematical Problem Solving With the MATH Datasetcited by 5,270
  8. 2016Asynchronous Methods for Deep Reinforcement Learningcited by 2,223
  9. 2024HybridFlow: A Flexible and Efficient RLHF Frameworkcited by 2,003
  10. 2024LiveCodeBench: Holistic and Contamination Free Evaluation of Large Language Models for Codecited by 1,951
  11. 2024OlympiadBench: A Challenging Benchmark for Promoting AGI with Olympiad-Level Bilingual Multimodal Scientific Problemscited by 1,111
  12. 2024SimPO: Simple Preference Optimization with a Reference-Free Rewardcited by 954
9 more
  1. 2025SFT Memorizes, RL Generalizes: A Comparative Study of Foundation Model Post-trainingcited by 663
  2. 2025rStar-Math: Small LLMs Can Master Math Reasoning with Self-Evolved Deep Thinkingcited by 309
  3. 2025ARC-AGI-2: A New Challenge for Frontier AI Reasoning Systemscited by 181
  4. 2025MedVLM-R1: Incentivizing Medical Reasoning Capability of Vision-Language Models (VLMs) via Reinforcement Learningcited by 173
  5. 2021Tianshou: a Highly Modularized Deep Reinforcement Learning Librarycited by 139
  6. 2025LLMs Can Easily Learn to Reason from Demonstrations Structure, not content, is what matters!cited by 132
  7. 2024Understanding the performance gap between online and offline alignment algorithmscited by 110
  8. 2024Critical Tokens Matter: Token-Level Contrastive Estimation Enhances LLM's Reasoning Capabilitycited by 81
  9. 2025ART: Anonymous Region Transformer for Variable Multi-Layer Transparent Image Generationcited by 37

Built on by

Cited by year (2026 to 13 Oct)

Citing papers per year: 2025: 216; 2026: 332.

548 papers.

Sort:

By area:

  1. 2025GLM-4.5: Agentic, Reasoning, and Coding (ARC) Foundation Modelscited by 456
  2. 2026Reinforcement Learning via Self-Distillationcited by 310
  3. 2025Reasoning with Exploration: An Entropy Perspectivecited by 279
  4. 2025MiniMax-M1: Scaling Test-Time Compute Efficiently with Lightning Attentioncited by 254
  5. 2025Spurious Rewards: Rethinking Training Signals in RLVRcited by 205
  6. 2025The Landscape of Agentic Reinforcement Learning for LLMs: A Surveycited by 200
  7. 2025A Survey of Reinforcement Learning for Large Reasoning Modelscited by 177
  8. 2025Agentic Reinforced Policy Optimizationcited by 168
  9. 2026Entropy-Aware On-Policy Distillation of Language Modelscited by 164
  10. 2025Scaling Latent Reasoning via Looped Language Modelscited by 153
  11. 2025Pass@k Training for Adaptively Balancing Exploration and Exploitation of Large Reasoning Modelscited by 136
  12. 2026Self-Distilled RLVRcited by 121
38 more
  1. 2025Geometric-Mean Policy Optimizationcited by 107
  2. 2025The Invisible Leash: Why RLVR May or May Not Escape Its Origincited by 95
  3. 2026TIP: Token Importance in On-Policy Distillationcited by 95
  4. 2025On-Policy RL Meets Off-Policy Experts: Harmonizing Supervised Fine-Tuning and Reinforcement Learning via Dynamic Weightingcited by 90
  5. 2025SRFT: A Single-Stage Method with Supervised and Reinforcement Fine-Tuning for Reasoningcited by 82
  6. 2026Scaling Reasoning Efficiently via Relaxed On-Policy Distillationcited by 74
  7. 2025Magistralcited by 59
  8. 2025Blending Supervised and Reinforcement Fine-Tuning with Prefix Samplingcited by 59
  9. 2025Parallel-R1: Towards Parallel Thinking via Reinforcement Learningcited by 59
  10. 2025e3: Learning to Explore Enables Extrapolation of Test-Time Compute for LLMscited by 57
  11. 2025rStar2-Agent: Agentic Reasoning Technical Reportcited by 54
  12. 2026CRISP: Compressed Reasoning via Iterative Self-Policy Distillationcited by 54
  13. 2025Rethinking Entropy Interventions in RLVR: An Entropy Change Perspectivecited by 52
  14. 2025Prosperity before Collapse: How Far Can Off-Policy RL Reach with Stale Data on LLMs?cited by 51
  15. 2025Intern-S1: A Scientific Multimodal Foundation Modelcited by 50
  16. 2025Scaf-GRPO: Scaffolded Group Relative Policy Optimization for Enhancing LLM Reasoningcited by 50
  17. 2025No Prompt Left Behind: Exploiting Zero-Variance Prompts in LLM Reinforcement Learning via Entropy-Guided Advantage Shapingcited by 49
  18. 2025BAPO: Stabilizing Off-Policy Reinforcement Learning for LLMs via Balanced Policy Optimization with Adaptive Clippingcited by 47
  19. 2026Fast and Effective On-policy Distillation from Reasoning Prefixescited by 47
  20. 2025TreeRPO: Tree Relative Policy Optimizationcited by 46
  21. 2026Not All Disagreement Is Learnable: Token Teachability in On-Policy Distillationcited by 46
  22. 2025First Return, Entropy-Eliciting Explorecited by 45
  23. 2025Harnessing Uncertainty: Entropy-Modulated Policy Gradients for Long-Horizon LLM Agentscited by 44
  24. 2025Evolving Language Models without Labels: Majority Drives Selection, Novelty Promotes Variationcited by 44
  25. 2025QuestA: Expanding Reasoning Capacity in LLMs via Question Augmentationcited by 42
  26. 2025Emergent Hierarchical Reasoning in LLMs through Reinforcement Learningcited by 42
  27. 2026POPE: Learning to Reason on Hard Problems via Privileged On-Policy Explorationcited by 40
  28. 2025Stabilizing Knowledge, Promoting Reasoning: Dual-Token Constraints for RLVRcited by 39
  29. 2026Embarrassingly Simple Self-Distillation Improves Code Generationcited by 39
  30. 2025RLEP: Reinforcement Learning with Experience Replay for LLM Reasoningcited by 38
  31. 2025DCPO: Dynamic Clipping Policy Optimizationcited by 38
  32. 2025FlowRL: Matching Reward Distributions for LLM Reasoningcited by 38
  33. 2025Spotlight on Token Perception for Multimodal Reinforcement Learningcited by 38
  34. 2025No Free Lunch: Rethinking Internal Feedback for LLM Reasoningcited by 36
  35. 2025Towards a Unified View of Large Language Model Post-Trainingcited by 36
  36. 2025The Choice of Divergence: A Neglected Key to Mitigating Diversity Collapse in Reinforcement Learning with Verifiable Rewardcited by 36
  37. 2025ETTRL: Balancing Exploration and Exploitation in LLM Test-Time Reinforcement Learning Via Entropy Mechanismcited by 35
  38. 2025Reinforcement Learning Meets Large Language Models: A Survey of Advancements and Applications Across the LLM Lifecyclecited by 35

50 most cited of 548 All 548 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2026Sparse but Critical: A Token-Level Analysis of Distributional Shifts in RLVR Fine-Tuning of LLMs
  2. 2025Not All Tokens Matter: Towards Efficient LLM Reasoning via Token Significance in Reinforcement Learning
  3. 2025Exploring Multi-Temperature Strategies for Token- and Rollout-Level Control in RLVR
  4. 2026Boosting LLM Exploration via Weak-Model Guidance in RLVR
  5. 2025The Entropy Mechanism of Reinforcement Learning for Reasoning Language Models
  6. 2025Reinforcement Learning with Verifiable Rewards Implicitly Incentivizes Correct Reasoning in Base LLMs
  7. 2026Entropy Is Not Enough: Unlocking Effective Reinforcement Learning for Visual Reasoning via Vision-Anchored Token Selection
  8. 2025The Debate on RLVR Reasoning Capability Boundary: Shrinkage, Expansion, or Both? A Two-Stage Dynamic View