# Self-Training Large Language Models with Confident Reasoning

arXiv: https://arxiv.org/abs/2505.17454 · HTML: https://mlsanity.com/paper/2505.17454/

## Key facts

- Cited by 8 papers in the map (data through 14 Oct 2026).
- Placed in Test-time compute and efficient thinking, one of 17 directions in LLM reasoning.
- Builds on 14 of its 46 references that are in the map.
- First posted on arXiv 23 May 2025.

## Placed in

- LLM reasoning / [Test-time compute and efficient thinking](https://mlsanity.com/lr/test-time-compute-and-efficient-thinking/)
- Synthetic data and self-improvement for LLMs / [Synthetic reasoning data and bootstrapping](https://mlsanity.com/synth/synthetic-reasoning-data-and-bootstrapping/)

## Builds on (most cited first, 14 of 14)

- [Training Verifiers to Solve Math Word Problems](https://mlsanity.com/paper/2110.14168/) (2021)
- [DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning](https://mlsanity.com/paper/2501.12948/) (2025)
- [Measuring Mathematical Problem Solving With the MATH Dataset](https://mlsanity.com/paper/2103.03874/) (2021)
- [Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challenge](https://mlsanity.com/paper/1803.05457/) (2018)
- [GPQA: A Graduate-Level Google-Proof Q&A Benchmark](https://mlsanity.com/paper/2311.12022/) (2023)
- [Language Models (Mostly) Know What They Know](https://mlsanity.com/paper/2207.05221/) (2022)
- [Measuring Faithfulness in Chain-of-Thought Reasoning](https://mlsanity.com/paper/2307.13702/) (2023)
- [Absolute Zero: Reinforced Self-play Reasoning with Zero Data](https://mlsanity.com/paper/2505.03335/) (2025)
- [TTRL: Test-Time Reinforcement Learning](https://mlsanity.com/paper/2504.16084/) (2025)
- [Direct Language Model Alignment from Online AI Feedback](https://mlsanity.com/paper/2402.04792/) (2024)
- [Confidence Improves Self-Consistency in LLMs](https://mlsanity.com/paper/2502.06233/) (2025)
- [Self-Consistency Preference Optimization](https://mlsanity.com/paper/2411.04109/) (2024)
- [Cycles of Thought: Measuring LLM Confidence through Stable Explanations](https://mlsanity.com/paper/2406.03441/) (2024)
- [Self-Refine Instruction-Tuning for Aligning Reasoning in Language Models](https://mlsanity.com/paper/2405.00402/) (2024)

## Built on by (most cited first, 8 of 8)

- [Teaching Language Models to Faithfully Express their Uncertainty](https://mlsanity.com/paper/2510.12587/) (2025)
- [On the Generalization Gap in Self-Evolving Language Model Reasoning](https://mlsanity.com/paper/2606.01075/) (2026)
- [ZeroSiam: An Efficient Asymmetry for Test-Time Entropy Optimization without Collapse](https://mlsanity.com/paper/2509.23183/) (2025)
- [Amplification Effects in Test-Time Reinforcement Learning: Safety and Reasoning Vulnerabilities](https://mlsanity.com/paper/2603.15417/) (2026)
- [Skill Weaving: Efficient LLM Improvement via Modular Skillpacks](https://mlsanity.com/paper/2605.22205/) (2026)
- [Reasoning Portability: Guiding Continual Learning for MLLMs in the RLVR Era](https://mlsanity.com/paper/2605.18903/) (2026)
- [Scaling LLM Reasoning from Minimal Labels: A Semi-Supervised Framework with a Lightweight Verifier](https://mlsanity.com/paper/2606.16811/) (2026)
- [Unsupervised Post-Training of Foundation Models: A Survey](https://mlsanity.com/paper/2608.24982/) (2026)

## Abstract (arXiv)

Large language models (LLMs) have shown impressive performance by generating reasoning paths before final answers, but learning such a reasoning path requires costly human supervision. To address this issue, recent studies have explored self-training methods that improve reasoning capabilities using pseudo-labels generated by the LLMs themselves. Among these, confidence-based self-training fine-tunes LLMs to prefer reasoning paths with high-confidence answers, where confidence is estimated via majority voting. However, such methods exclusively focus on the quality of the final answer and may ignore the quality of the reasoning paths, as even an incorrect reasoning path leads to a correct answer by chance. Instead, we advocate the use of reasoning-level confidence to identify high-quality reasoning paths for self-training, supported by our empirical observations. We then propose a new self-training method, CORE-PO, that fine-tunes LLMs to prefer high-COnfidence REasoning paths through Policy Optimization. Our experiments show that CORE-PO improves the accuracy of outputs on four in-distribution and two out-of-distribution benchmarks, compared to existing self-training methods.