mlsanityData to 14 Oct 2026

Uncalibrated Reasoning: GRPO Induces Overconfidence for Stochastic Outcomes

arXiv 2508.11800 ↗First posted 15 Aug 2025cs.LG cs.AIcited by 22

On the map: LLM reasoning → Reinforcement learning for reasoning

Abstract (arXiv)

Reinforcement learning (RL) has proven remarkably effective at improving the accuracy of language models in verifiable and deterministic domains like mathematics. Here, we examine if current RL methods are also effective at optimizing language models in verifiable domains with stochastic outcomes, like scientific experiments. Through applications to synthetic data and real-world biological experiments, we demonstrate that Group Relative Policy Optimization (GRPO) induces overconfident probability predictions for binary stochastic outcomes, while Proximal Policy Optimization (PPO) and REINFORCE Leave-One-Out (RLOO) yield well-calibrated models. We show that removing group standard normalization in GRPO fixes its miscalibration and provide a theoretical explanation for why normalization causes overconfidence. Our results provide new evidence against the use of standard normalization in GRPO and help pave the way for applications of RL for reasoning language models beyond deterministic domains.

Key facts. Cited by 22 papers in the map (data through 14 Oct 2026). Placed in Reinforcement learning for reasoning, one of 17 directions in LLM reasoning. Builds on 2 of its 19 references that are in the map. First posted on arXiv 15 Aug 2025.

Builds on

2 of 19 references.

References · 2

  1. 2025Qwen3 Technical Reportcited by 8,820
  2. 2024HybridFlow: A Flexible and Efficient RLHF Frameworkcited by 2,057

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 8; 2026: 14.

22 papers.

Sort:

By area:

  1. 2025OneThinker: All-in-one Reasoning Model for Image and Videocited by 56
  2. 2025Revisiting Entropy in Reinforcement Learning for Large Reasoning Modelscited by 21
  3. 2025Trace Length is a Simple Uncertainty Signal in Reasoning Modelscited by 19
  4. 2026Decoupling Reasoning and Confidence: Resurrecting Calibration in Reinforcement Learning from Verifiable Rewardscited by 14
  5. 2025DeepSearch: Overcome the Bottleneck of Reinforcement Learning with Verifiable Rewards via Monte Carlo Tree Searchcited by 11
  6. 2025C2GSPG: Confidence-calibrated Group Sequence Policy Gradient towards Self-aware Reasoningcited by 8
  7. 2026Meta-Reinforcement Learning with Self-Reflection for Agentic Searchcited by 8
  8. 2026Why GRPO Needs Normalization: A Local-Curvature Perspective on Adaptive Gradientscited by 5
  9. 2026Calibration-Aware Policy Optimization for Reasoning LLMscited by 5
  10. 2026The Dark Room in the Reward Channel: Dense Prediction Rewards Collapse GRPO-Trained LLM Agents -- and The Channel, Not the Content, Decides What Workscited by 5
  11. 2025EmoCaliber: Advancing Reliable Visual Emotion Comprehension via Confidence Verbalization and Calibrationcited by 4
  12. 2025Let the Model Distribute Its Doubt: Confidence Estimation through Verbalized Probability Distributioncited by 3
10 more
  1. 2026Power Reinforcement Post-Training of Text-to-Image Models with Super-Linear Advantage Shapingcited by 3
  2. 2026OpenVLThinkerV2: A Generalist Multimodal Reasoning Model for Multi-domain Visual Taskscited by 2
  3. 2026Process Supervision of Confidence Margin for Calibrated LLM Reasoningcited by 2
  4. 2026On the effectiveness of reward functions in reinforcement learning for confidence calibration of large language modelscited by 2
  5. 2026Multimodal Reward Hacking in Reinforcement Learningcited by 2
  6. 2025Beyond Token-level Supervision: Unlocking the Potential of Decoding-based Regression via Reinforcement Learningcited by 1
  7. 2026SWE-Fuse: Empowering Software Agents via Issue-free Trajectory Learning and Entropy-aware RLVR Trainingcited by 1
  8. 2026When Words Fall Short: Iterative Synergy Between Verbalized Reasoning and Hidden Features for LLM Confidence Estimationcited by 0
  9. 2026Beyond Verbalized Confidence: Calibrating Reasoners with Differentiable Readoutscited by 0
  10. 2026Test-time Calibration Learning for Large Language Model Reasoningcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Can GRPO Help LLMs Transcend Their Pretraining Origin?
  2. 2025CoRPO: Adding a Correctness Bias to GRPO Improves Generalization
  3. 2025Noise-corrected GRPO: From Noisy Rewards to Unbiased Gradients
  4. 2025Rewarding the Unlikely: Lifting GRPO Beyond Distribution Sharpening
  5. 2025Mitigating Think-Answer Mismatch in LLM Reasoning Through Noise-Aware Advantage Reweighting
  6. 2025Scaf-GRPO: Scaffolded Group Relative Policy Optimization for Enhancing LLM Reasoning
  7. 2026Revisiting Reinforcement Learning with Verifiable Rewards from a Contrastive Perspective
  8. 2026ReCo: Reweighting GRPO Against Distributional Concentration