mlsanityData to 14 Oct 2026

Unlocking Exploration in RLVR: Uncertainty-aware Advantage Shaping for Deeper Reasoning

arXiv 2510.10649 ↗First posted 12 Oct 2025cs.AIcited by 34

On the map: LLM reasoning → Reinforcement learning for reasoning

Abstract (arXiv)

Reinforcement Learning with Verifiable Rewards (RLVR) has shown significant promise for enhancing the reasoning capabilities of large language models (LLMs). However, prevailing algorithms like GRPO broadcast a uniform advantage signal across all tokens in a sequence. This coarse-grained approach overlooks the pivotal role of uncertain, high-stakes decisions during reasoning, leading to inefficient exploration and the well-documented problem of entropy collapse. To address this, we introduce UnCertainty-aware Advantage Shaping (UCAS), a model-free method that refines credit assignment by leveraging the model's internal uncertainty signals. UCAS operates in two stages: it first modulates the response-level advantage using a logit-space self-confidence proxy, and then applies an asymmetric token-level penalty based on raw logit certainty. This dual mechanism encourages exploration of high-uncertainty paths that yield correct answers while penalizing overconfident yet erroneous reasoning, effectively balancing the exploration-exploitation trade-off. Extensive experiments on five mathematical reasoning benchmarks show that UCAS significantly outperforms strong RLVR baselines across multiple model scales, including 1.5B and 7B. Our analysis confirms that UCAS not only achieves higher rewards but also promotes greater reasoning diversity and successfully mitigates entropy collapse. Code is available at https://github.com/xvolcano02/UCAS.

Key facts. Cited by 34 papers in the map (data through 14 Oct 2026). Placed in Reinforcement learning for reasoning, one of 17 directions in LLM reasoning. Builds on 34 of its 46 references that are in the map. First posted on arXiv 12 Oct 2025.

Builds on

34 of 46 references.

Closest ancestors · 20

Same direction, newest first

  1. 2025Rethinking Entropy Interventions in RLVR: An Entropy Change Perspective11 Oct
  2. 2025RL-PLUS: Countering Capability Boundary Collapse of LLMs in Reinforcement Learning with Hybrid-policy Optimization31 Jul
  3. 2025Reasoning with Exploration: An Entropy Perspective17 Jun
  4. 2025Beyond the 80/20 Rule: High-Entropy Minority Tokens Drive Effective Reinforcement Learning for LLM Reasoning2 Jun
  5. 2025The Entropy Mechanism of Reinforcement Learning for Reasoning Language Models28 May
  6. 2025KTAE: A Model-Free Algorithm to Key-Tokens Advantage Estimation in Mathematical Reasoning22 May
  7. 2025SEED-GRPO: Semantic Entropy Enhanced GRPO for Uncertainty-Aware Policy Optimization18 May
  8. 2025Reinforcement Learning for Reasoning in Large Language Models with One Training Example29 Apr
  9. 2025Does Reinforcement Learning Really Incentivize Reasoning Capacity in LLMs Beyond the Base Model?18 Apr
  10. 2025Open-Reasoner-Zero: An Open Source Approach to Scaling Up Reinforcement Learning on the Base Model31 Mar
  11. 2025Crossing the Reward Bridge: Expanding RL with Verifiable Rewards Across Diverse Domains31 Mar
  12. 2025Understanding R1-Zero-Like Training: A Critical Perspective26 Mar
8 more
  1. 2025SimpleRL-Zoo: Investigating and Taming Zero Reinforcement Learning for Open Base Models in the Wild24 Mar
  2. 2025DAPO: An Open-Source LLM Reinforcement Learning System at Scale18 Mar
  3. 2025An Efficient and Precise Training Data Construction Framework for Process-supervised Reward Model in Mathematical Reasoning4 Mar
  4. 2025Process Reinforcement through Implicit Rewards3 Feb
  5. 2024Generative Verifiers: Reward Modeling as Next-Token Prediction27 Aug
  6. 2024Improve Mathematical Reasoning in Language Models by Automated Process Supervision5 Jun
  7. 2023Math-Shepherd: Verify and Reinforce LLMs Step-by-step without Human Annotations14 Dec
  8. 2023Let's Verify Step by Step31 May

Other references · 14

  1. 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 9,379
  2. 2020Measuring Massive Multitask Language Understandingcited by 7,963
  3. 2022Self-Consistency Improves Chain of Thought Reasoning in Language Modelscited by 5,708
  4. 2021Measuring Mathematical Problem Solving With the MATH Datasetcited by 5,371
  5. 2024HybridFlow: A Flexible and Efficient RLHF Frameworkcited by 2,057
  6. 2024LiveCodeBench: Holistic and Contamination Free Evaluation of Large Language Models for Codecited by 2,004
  7. 2022Solving Quantitative Reasoning Problems with Language Modelscited by 1,392
  8. 2024DeepSeek-Coder: When the Large Language Model Meets Programming -- The Rise of Code Intelligencecited by 1,181
  9. 2024Qwen2.5-Math Technical Report: Toward Mathematical Expert Model via Self-Improvementcited by 1,090
  10. 2025Scalable Best-of-N Selection for Large Language Models via Self-Certaintycited by 187
  11. 2025Uncertainty Quantification and Confidence Calibration in Large Language Models: A Surveycited by 95
  12. 2024Step-level Value Preference Optimization for Mathematical Reasoningcited by 72
2 more
  1. 2024OlympicArena: Benchmarking Multi-discipline Cognitive Reasoning for Superintelligent AIcited by 71
  2. 2025The Choice of Divergence: A Neglected Key to Mitigating Diversity Collapse in Reinforcement Learning with Verifiable Rewardcited by 37

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 1; 2026: 33.

34 papers.

Sort:

By area:

  1. 2026Self-Distilled RLVRcited by 131
  2. 2025AMO-Bench: Large Language Models Still Struggle in High School Math Competitionscited by 38
  3. 2026Outcome-Grounded Advantage Reshaping for Fine-Grained Credit Assignment in Mathematical Reasoningcited by 19
  4. 2026Reinforcement Learning from Rich Feedback with Distributional DAggercited by 10
  5. 2026ATTNPO: Attention-Guided Process Supervision for Efficient Reasoningcited by 9
  6. 2026SELAUR: Self Evolving LLM Agent via Uncertainty-aware Rewardscited by 9
  7. 2026Reinforced Efficient Reasoning via Semantically Diverse Explorationcited by 8
  8. 2026FoE: Forest of Errors Makes the First Solution the Best in Large Reasoning Modelscited by 8
  9. 2026Self-Distilled Policy Gradientcited by 8
  10. 2026Not All Tokens See Equally: Perception-Grounded Policy Optimization for Large Vision-Language Modelscited by 7
  11. 2026One Refiner to Unlock Them All: Inference-Time Reasoning Elicitation via Reinforcement Query Refinementcited by 6
  12. 2026Epistemic Gain, Aleatoric Cost: Uncertainty Decomposition in Multi-Agent Debate for Math Reasoningcited by 5
22 more
  1. 2026Learning from Own Solutions: Self-Conditioned Credit Assignment for Reinforcement Learning with Verifiable Rewardscited by 4
  2. 2026Leveraging Error Diversity in Group Rollouts for Reinforcement Learningcited by 3
  3. 2026SAGE: Shaping Anchors for Guided Exploration in RLVR of LLMscited by 3
  4. 2026Discounted Beta-Bernoulli Reward Estimation for Sample-Efficient Reinforcement Learning with Verifiable Rewardscited by 2
  5. 2026AMR-SD: Asymmetric Meta-Reflective Self-Distillation for Token-Level Credit Assignmentcited by 2
  6. 2026GRAIL: Gradient-Reweighted Advantages for Reinforcement Learning with Verifiable Rewardscited by 2
  7. 2026EDGE: Experience-Distillation for Guided Exploration in Agentic Reinforcement Learningcited by 2
  8. 2026AWM: Answerable Working Memory for Long-Document VQA Agentscited by 2
  9. 2026Distributional Clarity: The Hidden Driver of RL-Friendliness in Large Language Modelscited by 1
  10. 2026Think in Latent Thoughts: A New Paradigm for Gloss-Free Sign Language Translationcited by 1
  11. 2026Confidence Laundering in Agent Systems: Why Uncertainty Needs a Latent Carriercited by 1
  12. 2026When Implausible Tokens Get Reinforced: Tail-Aware Credit Calibration for LLM Reinforcement Learningcited by 1
  13. 2026Miner:Mining Intrinsic Mastery for Data-Efficient RL in Large Reasoning Modelscited by 0
  14. 2026OGER: A Robust Offline-Guided Exploration Reward for Hybrid Reinforcement Learningcited by 0
  15. 2026Reasoning Portability: Guiding Continual Learning for MLLMs in the RLVR Eracited by 0
  16. 2026EAPO: Entropy-Driven Adaptive Positive-Negative Sample Weighting for Policy Optimization in Open-Ended QAcited by 0
  17. 2026Guidance Contrastive Token Credit Assignment for Discrete Policy Optimizationcited by 0
  18. 2026Customer-Agent: Overcoming Context Limitations in Ultra-Long Shopping Trajectories via Tool-Augmented Agents and RLVRcited by 0
  19. 2026Consilience for Verifier-Free Test-Time Scalingcited by 0
  20. 2026Surprising Success, Repeated Failure: Entropy-Guided Credit Assignment for Exploration in LLM Reasoningcited by 0
  21. 2026Advancing Entropy-Level Credit Assignment in RLVR via Proximal Entropy Policy Optimizationcited by 0
  22. 2026Semifactual Credit-Augmented Policy Optimizationcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026Step Potential Advantage Estimation: Harnessing Intermediate Confidence and Correctness for Efficient Mathematical Reasoning
  2. 2025Know When to Explore: Difficulty-Aware Certainty as a Guide for LLM Reinforcement Learning
  3. 2025Beyond High-Entropy Exploration: Correctness-Aware Low-Entropy Segment-Based Advantage Shaping for Reasoning LLMs
  4. 2025Exploration vs Exploitation: Rethinking RLVR through Clipping, Entropy, and Spurious Reward
  5. 2025From Trial-and-Error to Improvement: A Systematic Analysis of LLM Exploration Mechanisms in RLVR
  6. 2026Back to Basics: Revisiting Exploration in Reinforcement Learning for LLM Reasoning via Generative Probabilities
  7. 2026How You Begin is How You Reason: Driving Exploration in RLVR via Prefix-Tuned Priors
  8. 2025The Debate on RLVR Reasoning Capability Boundary: Shrinkage, Expansion, or Both? A Two-Stage Dynamic View