# Uncalibrated Reasoning: GRPO Induces Overconfidence for Stochastic Outcomes

arXiv: https://arxiv.org/abs/2508.11800 · HTML: https://mlsanity.com/paper/2508.11800/

## Key facts

- Cited by 22 papers in the map (data through 14 Oct 2026).
- Placed in Reinforcement learning for reasoning, one of 17 directions in LLM reasoning.
- Builds on 2 of its 19 references that are in the map.
- First posted on arXiv 15 Aug 2025.

## Placed in

- LLM reasoning / [Reinforcement learning for reasoning](https://mlsanity.com/lr/reinforcement-learning-for-reasoning/)

## Builds on (most cited first, 2 of 2)

- [Qwen3 Technical Report](https://mlsanity.com/paper/2505.09388/) (2025)
- [HybridFlow: A Flexible and Efficient RLHF Framework](https://mlsanity.com/paper/2409.19256/) (2024)

## Built on by (most cited first, 22 of 22)

- [OneThinker: All-in-one Reasoning Model for Image and Video](https://mlsanity.com/paper/2512.03043/) (2025)
- [Revisiting Entropy in Reinforcement Learning for Large Reasoning Models](https://mlsanity.com/paper/2511.05993/) (2025)
- [Trace Length is a Simple Uncertainty Signal in Reasoning Models](https://mlsanity.com/paper/2510.10409/) (2025)
- [Decoupling Reasoning and Confidence: Resurrecting Calibration in Reinforcement Learning from Verifiable Rewards](https://mlsanity.com/paper/2603.09117/) (2026)
- [DeepSearch: Overcome the Bottleneck of Reinforcement Learning with Verifiable Rewards via Monte Carlo Tree Search](https://mlsanity.com/paper/2509.25454/) (2025)
- [C2GSPG: Confidence-calibrated Group Sequence Policy Gradient towards Self-aware Reasoning](https://mlsanity.com/paper/2509.23129/) (2025)
- [Meta-Reinforcement Learning with Self-Reflection for Agentic Search](https://mlsanity.com/paper/2603.11327/) (2026)
- [Why GRPO Needs Normalization: A Local-Curvature Perspective on Adaptive Gradients](https://mlsanity.com/paper/2601.23135/) (2026)
- [Calibration-Aware Policy Optimization for Reasoning LLMs](https://mlsanity.com/paper/2604.12632/) (2026)
- [The Dark Room in the Reward Channel: Dense Prediction Rewards Collapse GRPO-Trained LLM Agents -- and The Channel, Not the Content, Decides What Works](https://mlsanity.com/paper/2607.21273/) (2026)
- [EmoCaliber: Advancing Reliable Visual Emotion Comprehension via Confidence Verbalization and Calibration](https://mlsanity.com/paper/2512.15528/) (2025)
- [Let the Model Distribute Its Doubt: Confidence Estimation through Verbalized Probability Distribution](https://mlsanity.com/paper/2511.14275/) (2025)
- [Power Reinforcement Post-Training of Text-to-Image Models with Super-Linear Advantage Shaping](https://mlsanity.com/paper/2605.10937/) (2026)
- [OpenVLThinkerV2: A Generalist Multimodal Reasoning Model for Multi-domain Visual Tasks](https://mlsanity.com/paper/2604.08539/) (2026)
- [Process Supervision of Confidence Margin for Calibrated LLM Reasoning](https://mlsanity.com/paper/2604.23333/) (2026)
- [On the effectiveness of reward functions in reinforcement learning for confidence calibration of large language models](https://mlsanity.com/paper/2607.04332/) (2026)
- [Multimodal Reward Hacking in Reinforcement Learning](https://mlsanity.com/paper/2607.09492/) (2026)
- [Beyond Token-level Supervision: Unlocking the Potential of Decoding-based Regression via Reinforcement Learning](https://mlsanity.com/paper/2512.06533/) (2025)
- [SWE-Fuse: Empowering Software Agents via Issue-free Trajectory Learning and Entropy-aware RLVR Training](https://mlsanity.com/paper/2603.07927/) (2026)
- [When Words Fall Short: Iterative Synergy Between Verbalized Reasoning and Hidden Features for LLM Confidence Estimation](https://mlsanity.com/paper/2609.34454/) (2026)
- [Beyond Verbalized Confidence: Calibrating Reasoners with Differentiable Readouts](https://mlsanity.com/paper/2609.34857/) (2026)
- [Test-time Calibration Learning for Large Language Model Reasoning](https://mlsanity.com/paper/2610.02695/) (2026)

## Abstract (arXiv)

Reinforcement learning (RL) has proven remarkably effective at improving the accuracy of language models in verifiable and deterministic domains like mathematics. Here, we examine if current RL methods are also effective at optimizing language models in verifiable domains with stochastic outcomes, like scientific experiments. Through applications to synthetic data and real-world biological experiments, we demonstrate that Group Relative Policy Optimization (GRPO) induces overconfident probability predictions for binary stochastic outcomes, while Proximal Policy Optimization (PPO) and REINFORCE Leave-One-Out (RLOO) yield well-calibrated models. We show that removing group standard normalization in GRPO fixes its miscalibration and provide a theoretical explanation for why normalization causes overconfidence. Our results provide new evidence against the use of standard normalization in GRPO and help pave the way for applications of RL for reasoning language models beyond deterministic domains.