mlsanityData to 7 Oct 2026

Do Not Let Low-Probability Tokens Over-Dominate in RL for LLMs

arXiv 2505.12929 ↗First posted 19 May 2025cs.CL cs.AI cs.LGcited by 43

On the map: LLM reasoning → Reinforcement learning for reasoning

Abstract (arXiv)

Reinforcement learning (RL) has become a cornerstone for enhancing the reasoning capabilities of large language models (LLMs), with recent innovations such as Group Relative Policy Optimization (GRPO) demonstrating exceptional effectiveness. In this study, we identify a critical yet underexplored issue in RL training: low-probability tokens disproportionately influence model updates due to their large gradient magnitudes. This dominance hinders the effective learning of high-probability tokens, whose gradients are essential for LLMs' performance but are substantially suppressed. To mitigate this interference, we propose two novel methods: Advantage Reweighting and Low-Probability Token Isolation (Lopti), both of which effectively attenuate gradients from low-probability tokens while emphasizing parameter updates driven by high-probability tokens. Our approaches promote balanced updates across tokens with varying probabilities, thereby enhancing the efficiency of RL training. Experimental results demonstrate that they substantially improve the performance of GRPO-trained LLMs, achieving up to a 46.2% improvement in K&K Logic Puzzle reasoning tasks. Our implementation is available at https://github.com/zhyang2226/AR-Lopti.

Key facts. Cited by 43 papers in the map (data through 7 Oct 2026). Placed in Reinforcement learning for reasoning, one of 17 directions in LLM reasoning. Builds on 44 of its 54 references that are in the map. First posted on arXiv 19 May 2025.

Builds on

44 of 54 references.

Closest ancestors · 15

Same direction, newest first

  1. 2025A Minimalist Approach to LLM Reasoning: from Rejection Sampling to Reinforce15 Apr
  2. 2025Efficient Reinforcement Finetuning via Adaptive Curriculum Learning7 Apr
  3. 2025VAPO: Efficient and Reliable Reinforcement Learning for Advanced Reasoning Tasks7 Apr
  4. 2025Open-Reasoner-Zero: An Open Source Approach to Scaling Up Reinforcement Learning on the Base Model31 Mar
  5. 2025Understanding R1-Zero-Like Training: A Critical Perspective26 Mar
  6. 2025SimpleRL-Zoo: Investigating and Taming Zero Reinforcement Learning for Open Base Models in the Wild24 Mar
  7. 2025DAPO: An Open-Source LLM Reinforcement Learning System at Scale18 Mar
  8. 2025Logic-RL: Unleashing LLM Reasoning with Rule-Based Reinforcement Learning20 Feb
  9. 2025DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning22 Jan
  10. 2025Kimi k1.5: Scaling Reinforcement Learning with LLMs22 Jan
  11. 2024AlphaMath Almost Zero: Process Supervision without Process6 May
  12. 2023Math-Shepherd: Verify and Reinforce LLMs Step-by-step without Human Annotations14 Dec
3 more
  1. 2023Let's Verify Step by Step31 May
  2. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Model29 May
  3. 2017Proximal Policy Optimization Algorithms20 Jul

Other references · 29

  1. 2017Attention Is All You Needcited by 59,932
  2. 2020Language Models are Few-Shot Learnerscited by 32,199
  3. 2023GPT-4 Technical Reportcited by 18,132
  4. 2022Training language models to follow instructions with human feedbackcited by 15,712
  5. 2024The Llama 3 Herd of Modelscited by 14,192
  6. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  7. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  8. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,302
  9. 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 8,851
  10. 2021Measuring Mathematical Problem Solving With the MATH Datasetcited by 5,203
  11. 2017Deep reinforcement learning from human preferencescited by 3,846
  12. 2023Qwen Technical Reportcited by 3,273
17 more
  1. 2024Gemini 1.5: Unlocking multimodal understanding across millions of tokens of contextcited by 3,175
  2. 2020Learning to summarize from human feedbackcited by 2,562
  3. 2019Fine-Tuning Language Models from Human Preferencescited by 2,124
  4. 2024Qwen2 Technical Reportcited by 2,076
  5. 2024HybridFlow: A Flexible and Efficient RLHF Frameworkcited by 1,980
  6. 2022Solving Quantitative Reasoning Problems with Language Modelscited by 1,359
  7. 2024OlympiadBench: A Challenging Benchmark for Promoting AGI with Olympiad-Level Bilingual Multimodal Scientific Problemscited by 1,099
  8. 2024SimPO: Simple Preference Optimization with a Reference-Free Rewardcited by 951
  9. 2023Qwen-Audio: Advancing Universal Audio Understanding via Unified Large-Scale Audio-Language Modelscited by 591
  10. 2024ORPO: Monolithic Preference Optimization without Reference Modelcited by 556
  11. 2024Contrastive Preference Optimization: Pushing the Boundaries of LLM Performance in Machine Translationcited by 390
  12. 2024Is DPO Superior to PPO for LLM Alignment? A Comprehensive Studycited by 262
  13. 2024From r to Q^*: Your Language Model is Secretly a Q-Functioncited by 242
  14. 2024Monte Carlo Tree Search Boosts Reasoning via Iterative Preference Learningcited by 238
  15. 2024DPO Meets PPO: Reinforced Token Optimization for RLHFcited by 126
  16. 2024On Memorization of Large Language Models in Logical Reasoningcited by 114
  17. 2024Learning Dynamics of LLM Finetuningcited by 93

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 13; 2026: 30.

43 papers.

Sort:

By area:

  1. 2025BAPO: Stabilizing Off-Policy Reinforcement Learning for LLMs via Balanced Policy Optimization with Adaptive Clippingcited by 47
  2. 2025Stabilizing Knowledge, Promoting Reasoning: Dual-Token Constraints for RLVRcited by 38
  3. 2025DLER: Doing Length pEnalty Right - Incentivizing More Intelligence per Token via Reinforcement Learningcited by 34
  4. 2025Low-probability Tokens Sustain Exploration in Reinforcement Learning with Verifiable Rewardcited by 25
  5. 2025Attention Illuminates LLM Reasoning: The Preplan-and-Anchor Rhythm Enables Fine-Grained Policy Optimizationcited by 23
  6. 2025GTPO: Stabilizing Group Relative Policy Optimization via Gradient and Entropy Controlcited by 18
  7. 2025GTPO and GRPO-S: Token and Sequence-Level Reward Shaping with Policy Entropycited by 18
  8. 2025Beyond the Sampled Token: Preserving Candidate Support in RLVRcited by 18
  9. 2026Exploratory Memory-Augmented LLM Agent via Hybrid On- and Off-Policy Optimizationcited by 17
  10. 2025The Debate on RLVR Reasoning Capability Boundary: Shrinkage, Expansion, or Both? A Two-Stage Dynamic Viewcited by 13
  11. 2026On the Direction of RLVR Updates for LLM Reasoning: Identification and Exploitationcited by 12
  12. 2026MASPO: Unifying Gradient Utilization, Probability Mass, and Signal Reliability for Robust and Sample-Efficient LLM Reasoningcited by 11
31 more
  1. 2026STAPO: Stabilizing Reinforcement Learning for LLMs by Silencing Rare Spurious Tokenscited by 9
  2. 2025C2GSPG: Confidence-calibrated Group Sequence Policy Gradient towards Self-aware Reasoningcited by 8
  3. 2026On the Entropy Dynamics in Reinforcement Fine-Tuning of Large Language Modelscited by 8
  4. 2026Understanding Reasoning in LLMs through Strategic Information Allocation under Uncertaintycited by 8
  5. 2026Where Hindsight Credit Can Reside: A Signed-Capacity View of Token Updates in RLVRcited by 8
  6. 2026Reasoning-Driven Multimodal LLM for Domain Generalizationcited by 7
  7. 2026Advantage Collapse in Group Relative Policy Optimization: Diagnosis and Mitigationcited by 7
  8. 2026From log π to π: Taming Divergence in Soft Clipping via Bilateral Decoupled Decay of Probability Gradient Weightcited by 6
  9. 2025Stabilizing Policy Gradients for Sample-Efficient Reinforcement Learning in LLM Reasoningcited by 5
  10. 2026GRPO-VPS: Enhancing Group Relative Policy Optimization with Verifiable Process Supervision for Effective Reasoningcited by 5
  11. 2025Sharpness-Guided Group Relative Policy Optimization via Probability Shapingcited by 4
  12. 2026ReMiT: RL-Guided Mid-Training for Iterative LLM Evolutioncited by 4
  13. 2026How Does Reasoning Flow? Tracing Attention-Induced Information Flow for Targeted RL in LLMscited by 3
  14. 2026STARE: Surprisal-Guided Token-Level Advantage Reweighting for Policy Entropy Stabilitycited by 3
  15. 2026Which Tokens Matter? Adaptive Token Selection for RLVR with the Relative Surprisal Indexcited by 3
  16. 2026CoEvoKG: Co-Evolving Knowledge Graphs with Self-Evolving Search Agentscited by 3
  17. 2025ISOPO: Proximal policy gradients without pi-oldcited by 1
  18. 2026Online Causal Kalman Filtering for Stable and Effective Policy Optimizationcited by 1
  19. 2026Are complicated loss functions necessary for teaching LLMs to reason?cited by 1
  20. 2026A First-Principles Derivation of LLM Policy Optimization: From Expected Reward to GRPO and Its Structural Extensionscited by 1
  21. 2026Beyond Entropy: Learning from Token-Level Distributional Deviations for LLM Reasoningcited by 1
  22. 2026When Implausible Tokens Get Reinforced: Tail-Aware Credit Calibration for LLM Reinforcement Learningcited by 1
  23. 2026GMTS: Gradient Magnitude-based Token Selection Improves RLVR Training for LLM Reasoningcited by 1
  24. 2026A Survey of Reinforcement Learning for Large Language Models under Data Scarcity: Challenges and Solutionscited by 0
  25. 2026Holder Policy Optimisationcited by 0
  26. 2026When to Stop Reusing: Dynamic Gradient Gating for Sample-Efficient RLVRcited by 0
  27. 2026Clipping Bottleneck: Stabilizing RLVR via Stochastic Recovery of Near-Boundary Signalscited by 0
  28. 2026Not only where, But when: Temporal Scheduling for RLVRcited by 0
  29. 2026What are Key Factors for Updates in RL for LLM Reasoning?cited by 0
  30. 2026DataFlex-RL: An Evaluation Platform for RLVR Data Policiescited by 0
  31. 2026TISD: On-Policy Self-Distillation with Trajectory Interventioncited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Token-Efficient RL for LLM Reasoning
  2. 2026Rethinking RL for LLM Reasoning: It's Sparse Policy Selection, Not Capability Learning
  3. 2025Diversity or Precision? A Deep Dive into Next Token Prediction
  4. 2025Accelerating RL for LLM Reasoning with Optimal Advantage Regression
  5. 2025Scaling Up RL: Unlocking Diverse Reasoning in LLMs via Prolonged Training
  6. 2026A Step Back: Prefix Importance Ratio Stabilizes Policy Optimization
  7. 2026The Optimal Token Baseline: Variance Reduction for Long-Horizon LLM-RL
  8. 2025Not All Tokens Matter: Towards Efficient LLM Reasoning via Token Significance in Reinforcement Learning