mlsanityData to 14 Oct 2026

Beyond Entropy: Learning from Token-Level Distributional Deviations for LLM Reasoning

arXiv 2606.19771 ↗First posted 18 Jun 2026cs.AIcited by 1

On the map: LLM reasoning → Reinforcement learning for reasoning

Abstract (arXiv)

Reinforcement Learning with Verifiable Rewards (RLVR) has significantly advanced Large Language Model (LLM) reasoning; however, it faces a fundamental optimization instability: uniform token updates precipitate entropy collapse, leading to premature convergence to suboptimal strategies, whereas excessive Shannon Entropy maximization can cause entropy explosion, driving blind exploration toward incoherent reasoning chains. To resolve this dichotomy, we introduce the Independent Combinatorial Tokens (ICT) framework, which shifts the optimization focus from scalar uncertainty to the distributional properties of token logits. By leveraging the Jensen-Shannon (JS) divergence between token logits distributions, ICT identifies tokens with distinctive distributional patterns as critical branching points for guiding effective exploration in LLM reasoning. Our theoretical analysis, grounded in both Shannon and second-order R'enyi entropy, proves that selectively updating on these tokens regulates policy concentration: it reduces the overall distribution uncertainty measured by Shannon entropy, while controlling probability concentration captured by second-order R'enyi entropy. This dual effect prevents over-concentrated token generation from weakening exploration and effectively stabilizes the training landscape. Empirical results demonstrate that updating only the top 10% of unique tokens on Qwen2.5 (0.5B/1.5B/7B) models yields an average pass@4 improvement of 4.58%, with a maximum gain of 14.9%, over GRPO, 20-Entropy, and STAPO baselines across seven benchmarks spanning math, commonsense, and Olympiad-level problems.

Key facts. Cited by 1 paper in the map (data through 14 Oct 2026). Placed in Reinforcement learning for reasoning, one of 17 directions in LLM reasoning. Builds on 29 of its 48 references that are in the map. First posted on arXiv 18 Jun 2026.

Builds on

29 of 48 references.

Closest ancestors · 16

Same direction, newest first

  1. 2026Sparse but Critical: A Token-Level Analysis of Distributional Shifts in RLVR Fine-Tuning of LLMs23 Mar
  2. 2026STAPO: Stabilizing Reinforcement Learning for LLMs by Silencing Rare Spurious Tokens17 Feb
  3. 2025Efficient Reinforcement Learning with Semantic and Token Entropy for LLM Reasoning4 Dec
  4. 2025Revisiting Entropy in Reinforcement Learning for Large Reasoning Models8 Nov
  5. 2025Revisiting Entropy Regularization: Adaptive Coefficient Unlocks Its Potential for LLM Reinforcement Learning13 Oct
  6. 2025Rethinking Entropy Interventions in RLVR: An Entropy Change Perspective11 Oct
  7. 2025SCOPE-RL: Stable and Quantitative Control of Policy Entropy in RL Post-Training9 Oct
  8. 2025Rethinking Entropy Regularization in Large Reasoning Models29 Sep
  9. 2025Beyond the 80/20 Rule: High-Entropy Minority Tokens Drive Effective Reinforcement Learning for LLM Reasoning2 Jun
  10. 2025KTAE: A Model-Free Algorithm to Key-Tokens Advantage Estimation in Mathematical Reasoning22 May
  11. 2025AceReason-Nemotron: Advancing Math and Code Reasoning through Reinforcement Learning22 May
  12. 2025Do Not Let Low-Probability Tokens Over-Dominate in RL for LLMs19 May
4 more
  1. 2025Reinforcement Learning for Reasoning in Large Language Models with One Training Example29 Apr
  2. 2025DAPO: An Open-Source LLM Reinforcement Learning System at Scale18 Mar
  3. 2025DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning22 Jan
  4. 2023Let's Verify Step by Step31 May

Other references · 13

  1. 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 9,379
  2. 2020Measuring Massive Multitask Language Understandingcited by 7,963
  3. 2023GPQA: A Graduate-Level Google-Proof Q&A Benchmarkcited by 2,879
  4. 2024OpenAI o1 System Cardcited by 1,244
  5. 2024DeepSeek-V3 Technical Reportcited by 811
  6. 2023Safe RLHF: Safe Reinforcement Learning from Human Feedbackcited by 656
  7. 2024Critical Tokens Matter: Token-Level Contrastive Estimation Enhances LLM's Reasoning Capabilitycited by 83
  8. 2025Demystifying Reasoning Dynamics with Mutual Information: Thinking Tokens are Information Peaks in LLM Reasoningcited by 51
  9. 2025Reinforcement Learning Meets Large Language Models: A Survey of Advancements and Applications Across the LLM Lifecyclecited by 35
  10. 2025Enhancing Large Language Model Reasoning via Selective Critical Token Fine-Tuningcited by 20
  11. 2025Top-H Decoding: Adapting the Creativity and Coherence with Bounded Entropy in Text Generationcited by 9
  12. 2025Less is More: Improving LLM Reasoning with Minimal Test-Time Interventioncited by 5
1 more
  1. 2025Entropy-Guided Token Dropout: Training Autoregressive Language Models with Limited Domain Datacited by 5

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 1.

1 paper.

  1. 2026GRPODropout: Less is More for Online Reinforcement Learning Rolloutscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Beyond High-Entropy Exploration: Correctness-Aware Low-Entropy Segment-Based Advantage Shaping for Reasoning LLMs
  2. 2026GMTS: Gradient Magnitude-based Token Selection Improves RLVR Training for LLM Reasoning
  3. 2025Reasoning with Exploration: An Entropy Perspective
  4. 2025Low-probability Tokens Sustain Exploration in Reinforcement Learning with Verifiable Reward
  5. 2025Not All Tokens Matter: Towards Efficient LLM Reasoning via Token Significance in Reinforcement Learning
  6. 2025Diversity or Precision? A Deep Dive into Next Token Prediction
  7. 2026Not All Tokens Learn Alike: Attention Entropy Reveals Heterogeneous Signals in RL Reasoning
  8. 2026Taming the Thinker: Conditional Entropy Shaping for Adaptive LLM Reasoning