mlsanityData to 14 Oct 2026

Process Supervision of Confidence Margin for Calibrated LLM Reasoning

arXiv 2604.23333 ↗First posted 25 Apr 2026cs.LG cs.CLcited by 2

On the map: LLM reasoning → Reinforcement learning for reasoning

Abstract (arXiv)

Scaling test-time computation with reinforcement learning (RL) has emerged as a reliable path to improve large language models (LLM) reasoning ability. Yet, outcome-based reward often incentivizes models to be overconfident, leading to hallucinations, unreliable confidence-based control, and unnecessary compute allocation. We introduce Reinforcement Learning with Confidence Margin (RLCM), a calibration-aware RL framework that jointly optimizes correctness and confidence reliability via a margin-enhanced process reward over intermediate-budget completions. Rather than aligning confidence to correctness likelihoods, RLCM encourages the model to widen the confidence margin between correct and incorrect steps within a single reasoning trajectory. Across mathematical, code, logic and science benchmarks, our method substantially improves calibration while maintaining or improving accuracy. We further show that, with calibrated confidence signals, the resulting models enable more efficient conformal risk control and effective confidence-weighted aggregation

Key facts. Cited by 2 papers in the map (data through 14 Oct 2026). Placed in Reinforcement learning for reasoning, one of 17 directions in LLM reasoning. Builds on 75 of its 89 references that are in the map. First posted on arXiv 25 Apr 2026.

Builds on

75 of 89 references.

Closest ancestors · 18

Same direction, newest first

  1. 2026Decoupling Reasoning and Confidence: Resurrecting Calibration in Reinforcement Learning from Verifiable Rewards10 Mar
  2. 2025A Survey of Process Reward Models: From Outcome Signals to Process Supervisions for Large Language Models9 Oct
  3. 2025C2GSPG: Confidence-calibrated Group Sequence Policy Gradient towards Self-aware Reasoning27 Sep
  4. 2025Uncalibrated Reasoning: GRPO Induces Overconfidence for Stochastic Outcomes15 Aug
  5. 2025Beyond Binary Rewards: Training LMs to Reason About Their Uncertainty22 Jul
  6. 2025On the Design of KL-Regularized Policy Gradient Algorithms for LLM Reasoning23 May
  7. 2025GRPO-LEAD: A Difficulty-Aware Reinforcement Learning Approach for Concise Mathematical Reasoning in Language Models13 Apr
  8. 2025Synthetic Data Generation & Multi-Step RL for Reasoning & Tool Use7 Apr
  9. 2025DAPO: An Open-Source LLM Reinforcement Learning System at Scale18 Mar
  10. 2025Optimizing Test-Time Compute via Meta Reinforcement Fine-Tuning10 Mar
  11. 2025The Lessons of Developing Process Reward Models in Mathematical Reasoning13 Jan
  12. 2025Enhancing Reasoning through Process Supervision with Monte Carlo Tree Search2 Jan
6 more
  1. 2024Tulu 3: Pushing Frontiers in Open Language Model Post-Training22 Nov
  2. 2024Rewarding Progress: Scaling Automated Process Verifiers for LLM Reasoning10 Oct
  3. 2024Improve Mathematical Reasoning in Language Models by Automated Process Supervision5 Jun
  4. 2023Math-Shepherd: Verify and Reinforce LLMs Step-by-step without Human Annotations14 Dec
  5. 2023Let's Verify Step by Step31 May
  6. 2017Proximal Policy Optimization Algorithms20 Jul

Other references · 57

  1. 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 9,379
  2. 2021Measuring Mathematical Problem Solving With the MATH Datasetcited by 5,371
  3. 2017On Calibration of Modern Neural Networkscited by 4,960
  4. 2023GPQA: A Graduate-Level Google-Proof Q&A Benchmarkcited by 2,879
  5. 2024HybridFlow: A Flexible and Efficient RLHF Frameworkcited by 2,057
  6. 2024LiveCodeBench: Holistic and Contamination Free Evaluation of Large Language Models for Codecited by 2,004
  7. 2023A Survey on Hallucination in Large Language Models: Principles, Taxonomy, Challenges, and Open Questionscited by 1,601
  8. 2021Calibrate Before Use: Improving Few-Shot Performance of Language Modelscited by 1,392
  9. 2025s1: Simple test-time scalingcited by 1,341
  10. 2024OlympiadBench: A Challenging Benchmark for Promoting AGI with Olympiad-Level Bilingual Multimodal Scientific Problemscited by 1,136
  11. 2023Can LLMs Express Their Uncertainty? An Empirical Evaluation of Confidence Elicitation in LLMscited by 974
  12. 2023On-Policy Distillation of Language Models: Learning from Self-Generated Mistakescited by 939
38 more
  1. 2023Just Ask for Calibration: Strategies for Eliciting Calibrated Confidence Scores from Language Models Fine-Tuned with Human Feedbackcited by 812
  2. 2026Self-Distilled Reasoner: On-Policy Self-Distillation for Large Language Modelscited by 455
  3. 2026Reinforcement Learning via Self-Distillationcited by 332
  4. 2020Calibration of Pre-trained Transformerscited by 315
  5. 2022Conformal Risk Controlcited by 299
  6. 2021Learn then Test: Calibrating Predictive Algorithms to Achieve Risk Controlcited by 271
  7. 2024Semantic Entropy Probes: Robust and Cheap Hallucination Detection in LLMscited by 259
  8. 2025Why Language Models Hallucinatecited by 220
  9. 2025Scalable Best-of-N Selection for Large Language Models via Self-Certaintycited by 187
  10. 2023Conformal Language Modelingcited by 170
  11. 2025Reasoning Models Know When They're Right: Probing Hidden States for Self-Verificationcited by 159
  12. 2024Language Model Cascades: Token-level uncertainty and beyondcited by 121
  13. 2024Taming Overconfidence in LLMs: Reward Calibration in RLHFcited by 112
  14. 2024Large Language Models Must Be Taught to Know What They Don't Knowcited by 100
  15. 2025Uncertainty Quantification and Confidence Calibration in Large Language Models: A Surveycited by 95
  16. 2024Can Large Language Models Faithfully Express Their Intrinsic Uncertainty in Words?cited by 87
  17. 2023OVM, Outcome-supervised Value Models for Planning in Mathematical Reasoningcited by 85
  18. 2025Efficient Test-Time Scaling via Self-Calibrationcited by 75
  19. 2023Do Androids Know They're Only Dreaming of Electric Sheep?cited by 67
  20. 2024ConU: Conformal Uncertainty in Large Language Models with Correctness Coverage Guaranteescited by 65
  21. 2024Linguistic Calibration of Long-Form Generationscited by 60
  22. 2024When Do LLMs Need Retrieval Augmentation? Mitigating LLMs' Overconfidence Helps Retrieval Augmentationcited by 59
  23. 2022A Close Look into the Calibration of Pre-trained Language Modelscited by 57
  24. 2025Rewarding Doubt: A Reinforcement Learning Approach to Calibrated Confidence Expression of Large Language Modelscited by 52
  25. 2025Demystifying Reasoning Dynamics with Mutual Information: Thinking Tokens are Information Peaks in LLM Reasoningcited by 51
  26. 2025Mind the Confidence Gap: Overconfidence, Calibration, and Distractor Effects in Large Language Modelscited by 40
  27. 2025ConfTuner: Training Large Language Models to Express Their Confidence Verballycited by 39
  28. 2024Thermometer: Towards Universal Calibration for Large Language Modelscited by 38
  29. 2025Reasoning about Uncertainty: Do Reasoning Models Know When They Don't Know?cited by 38
  30. 2019Uncertain Natural Language Inferencecited by 35
  31. 2025Beyond Correctness: Harmonizing Process and Outcome Rewards through RL Trainingcited by 29
  32. 2024Evaluating language models as risk scorescited by 27
  33. 2025Optimizing Anytime Reasoning via Budget Relative Policy Optimizationcited by 27
  34. 2025LLM Probability Concentration: How Alignment Shrinks the Generative Horizoncited by 26
  35. 2025Thought calibration: Efficient and confident test-time scalingcited by 21
  36. 2025Conformal Linguistic Calibration: Trading-off between Factuality and Specificitycited by 20
  37. 2025Your Pre-trained LLM is Secretly an Unsupervised Confidence Calibratorcited by 20
  38. 2025Is That Your Final Answer? Test-Time Scaling Improves Selective Question Answeringcited by 17

50 most cited of 57 All 57 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 2.

2 papers.

  1. 2026On the Pitfalls of Verbalized Confidence Priors for Calibrating Large Reasoning Modelscited by 0
  2. 2026Learning More by Training Less: Uncertainty-Guided Progressive Contraction for Efficient Reasoning Post-Trainingcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026Scaling with Confidence: Calibrating Confidence of LLMs for Adaptive Test Time Scaling
  2. 2026Balancing Classification and Calibration Performance in Decision-Making LLMs via Calibration Aware Reinforcement Learning
  3. 2026RL-ARC: Calibrating Large Reasoning Models via Reasoning-guided Uncertainty
  4. 2026On the effectiveness of reward functions in reinforcement learning for confidence calibration of large language models
  5. 2026Shared Experience, Separate Learning: Companion Confidence Calibration for LLMs
  6. 2026ConSteer-RL: Steering Reasoning Capabilities in Large Language Models via Confidence-Aware Reinforcement Learning
  7. 2025ConfClip: Confidence-Weighted and Clipped Reward for Reinforcement Learning in LLMs
  8. 2026Understanding and Mitigating Premature Confidence for Better LLM Reasoning