Confidence-Aware Alignment Makes Reasoning LLMs More Reliable
On the map: LLM reasoning → Test-time compute and efficient thinking
Abstract (arXiv)
Large reasoning models often reach correct answers through flawed intermediate steps, creating a gap between final accuracy and reasoning reliability. Existing alignment strategies address this with external verifiers or massive sampling, limiting scalability. In this work, we introduce CASPO (Confidence-Aware Step-wise Preference Optimization), a framework that aligns token-level confidence with step-wise logical correctness through iterative Direct Preference Optimization, without training a separate reward model. During inference, we propose Confidence-aware Thought (CaT), which leverages this calibrated confidence to dynamically prune uncertain reasoning branches with negligible O(V) latency. Experiments across ten benchmarks and multiple model families show that CASPO consistently improves reasoning reliability and inference efficiency. CASPO scales to Qwen3-8B-Base and surpasses tree-search baselines on AIME'24 and AIME'25 without using reward-model data. We also release a step-wise dataset with confidence annotations to support fine-grained analysis of reasoning reliability. Code is available at https://github.com/Thecommonirin/CASPO.
Key facts. Cited by 1 paper in the map (data through 14 Oct 2026). Placed in Test-time compute and efficient thinking, one of 17 directions in LLM reasoning. Builds on 40 of its 59 references that are in the map. First posted on arXiv 8 May 2026.
Builds on
40 of 59 references.
Closest ancestors · 2
Same direction, newest first
- 2025CER: Confidence Enhanced Reasoning in LLMs20 Feb
- 2024Forest-of-Thought: Scaling Test-Time Compute for Enhancing LLM Reasoning12 Dec
Other references · 38
- 2024The Llama 3 Herd of Modelscited by 14,715
- 2021Training Verifiers to Solve Math Word Problemscited by 10,109
- 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 9,379
- 2025Qwen3 Technical Reportcited by 8,820
- 2021Evaluating Large Language Models Trained on Codecited by 8,320
- 2020Measuring Massive Multitask Language Understandingcited by 7,963
- 2025DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learningcited by 6,330
- 2022Self-Consistency Improves Chain of Thought Reasoning in Language Modelscited by 5,708
- 2021Measuring Mathematical Problem Solving With the MATH Datasetcited by 5,371
- 2024Qwen2.5 Technical Reportcited by 4,942
- 2024LiveCodeBench: Holistic and Contamination Free Evaluation of Large Language Models for Codecited by 2,004
- 2024OpenAI o1 System Cardcited by 1,244
26 more
- 2024OlympiadBench: A Challenging Benchmark for Promoting AGI with Olympiad-Level Bilingual Multimodal Scientific Problemscited by 1,136
- 2023Math-Shepherd: Verify and Reinforce LLMs Step-by-step without Human Annotationscited by 928
- 2017RACE: Large-scale ReAding Comprehension Dataset From Examinationscited by 861
- 2025SimpleRL-Zoo: Investigating and Taming Zero Reinforcement Learning for Open Base Models in the Wildcited by 511
- 2025Open-Reasoner-Zero: An Open Source Approach to Scaling Up Reinforcement Learning on the Base Modelcited by 461
- 2025From System 1 to System 2: A Survey of Reasoning Large Language Modelscited by 320
- 2025rStar-Math: Small LLMs Can Master Math Reasoning with Self-Evolved Deep Thinkingcited by 313
- 2024CRUXEval: A Benchmark for Code Reasoning, Understanding and Executioncited by 300
- 2025R1-Zero's "Aha Moment" in Visual Reasoning on a 2B Non-SFT Modelcited by 165
- 2024Fact-Checking the Output of Large Language Models via Token-Level Uncertainty Quantificationcited by 162
- 2025CPPO: Accelerating the Training of Group Relative Policy Optimization-Based Reasoning Modelscited by 96
- 2024Learning Dynamics of LLM Finetuningcited by 95
- 2023SAC3: Reliable Hallucination Detection in Black-Box Language Models via Semantic-aware Cross-check Consistencycited by 90
- 2024OpenRLHF: An Easy-to-use, Scalable and High-performance RLHF Frameworkcited by 73
- 2025TreePO: Bridging the Gap of Policy Optimization and Efficacy and Inference Efficiency with Heuristic Tree-based Modelingcited by 61
- 2024GenARM: Reward Guided Generation with Autoregressive Reward Model for Test-time Alignmentcited by 54
- 2025Satori: Reinforcement Learning with Chain-of-Action-Thought Enhances LLM Reasoning via Autoregressive Searchcited by 53
- 2025Stop Summation: Min-Form Credit Assignment Is All Process Reward Model Needs for Reasoningcited by 50
- 2025SwS: Self-aware Weakness-driven Problem Synthesis in Reinforcement Learning for LLM Reasoningcited by 42
- 2025Enhancing LLM Reasoning with Iterative DPO: A Comprehensive Empirical Investigationcited by 37
- 2025Process-based Self-Rewarding Language Modelscited by 22
- 2025Towards Safe Reasoning in Large Reasoning Models via Corrective Interventioncited by 11
- 2024DiPT: Enhancing LLM reasoning through diversified perspective-takingcited by 4
- 2025Learning Together to Perform Better: Teaching Small-Scale LLMs to Collaborate via Preferential Rationale Tuningcited by 3
- 2025Probability-Consistent Preference Optimization for Enhanced LLM Reasoningcited by 2
- 2025It Helps to Take a Second Opinion: Teaching Smaller LLMs to Deliberate Mutually via Selective Rationale Optimisationcited by 1
Built on by
Citing papers per year: 2026: 1.
1 paper.
Similar, not cited
Close in wording, no citation either way
- 2026CoRA: Confidence-Rationale Alignment for Reliable Chain-of-Thought Reasoning
- 2026CAT: Confidence-Adaptive Thinking for Efficient Reasoning of Large Reasoning Models
- 2026Unsupervised Confidence Calibration for Reasoning LLMs from a Single Generation
- 2025Confidence Improves Self-Consistency in LLMs
- 2026Beyond Confidence: Stability-Aware Test-Time Adaptation for LLM Reasoning
- 2026Quantifying Faithful Confidence Expression in Large Reasoning Models
- 2026Understanding and Mitigating Premature Confidence for Better LLM Reasoning
- 2025Think Just Enough: Sequence-Level Entropy as a Confidence Signal for LLM Reasoning