# From the Inside Out: Progressive Distribution Refinement for Confidence Calibration

arXiv: https://arxiv.org/abs/2603.16500 · HTML: https://mlsanity.com/paper/2603.16500/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Placed in Reward hacking and misspecification, one of 10 directions in LLM alignment / RLHF / preference optimisation.
- Builds on 24 of its 32 references that are in the map.
- First posted on arXiv 17 Mar 2026.

## Placed in

- LLM alignment / RLHF / preference optimisation / [Reward hacking and misspecification](https://mlsanity.com/alignment/reward-hacking-and-misspecification/)

## Builds on (most cited first, 24 of 24)

- [Chain-of-Thought Prompting Elicits Reasoning in Large Language Models](https://mlsanity.com/paper/2201.11903/) (2022)
- [Qwen3 Technical Report](https://mlsanity.com/paper/2505.09388/) (2025)
- [Self-Consistency Improves Chain of Thought Reasoning in Language Models](https://mlsanity.com/paper/2203.11171/) (2022)
- [Measuring Mathematical Problem Solving With the MATH Dataset](https://mlsanity.com/paper/2103.03874/) (2021)
- [Let's Verify Step by Step](https://mlsanity.com/paper/2305.20050/) (2023)
- [GPQA: A Graduate-Level Google-Proof Q&A Benchmark](https://mlsanity.com/paper/2311.12022/) (2023)
- [Does Reinforcement Learning Really Incentivize Reasoning Capacity in LLMs Beyond the Base Model?](https://mlsanity.com/paper/2504.13837/) (2025)
- [Self-supervised Learning: Generative or Contrastive](https://mlsanity.com/paper/2006.08218/) (2020)
- [Titans: Learning to Memorize at Test Time](https://mlsanity.com/paper/2501.00663/) (2024)
- [MathArena: Evaluating LLMs on Uncontaminated Math Competitions](https://mlsanity.com/paper/2505.23281/) (2025)
- ["Zero-Shot" Super-Resolution using Deep Internal Learning](https://mlsanity.com/paper/1712.06087/) (2017)
- [TTRL: Test-Time Reinforcement Learning](https://mlsanity.com/paper/2504.16084/) (2025)
- [A Survey of Confidence Estimation and Calibration in Large Language Models](https://mlsanity.com/paper/2311.08298/) (2023)
- [Deep Think with Confidence](https://mlsanity.com/paper/2508.15260/) (2025)
- [Scalable Best-of-N Selection for Large Language Models via Self-Certainty](https://mlsanity.com/paper/2502.18581/) (2025)
- [Semantic Photo Manipulation with a Generative Image Prior](https://mlsanity.com/paper/2005.07727/) (2020)
- [Fact-Checking the Output of Large Language Models via Token-Level Uncertainty Quantification](https://mlsanity.com/paper/2403.04696/) (2024)
- [Right Question is Already Half the Answer: Fully Unsupervised LLM Reasoning Incentivization](https://mlsanity.com/paper/2504.05812/) (2025)
- [The Surprising Effectiveness of Test-Time Training for Few-Shot Learning](https://mlsanity.com/paper/2411.07279/) (2024)
- [When to Trust LLMs: Aligning Confidence with Response Quality](https://mlsanity.com/paper/2404.17287/) (2024)
- [Generalist Reward Models: Found Inside Large Language Models](https://mlsanity.com/paper/2506.23235/) (2025)
- [Beyond Majority Voting: Towards Fine-grained and More Reliable Reward Signal for Test-Time Reinforcement Learning](https://mlsanity.com/paper/2512.15146/) (2025)
- [Majority of the Bests: Improving Best-of-N via Bootstrapping](https://mlsanity.com/paper/2511.18630/) (2025)
- [Believe Your Model: Distribution-Guided Confidence Calibration](https://mlsanity.com/paper/2603.03872/) (2026)

## Abstract (arXiv)

Leveraging the model's internal information as the self-reward signal in Reinforcement Learning (RL) has received extensive attention due to its label-free nature. While prior works have made significant progress in applying the Test-Time Scaling (TTS) strategies to RL, the discrepancy in internal information between test and training remains inadequately addressed. Moreover, Test-Time Training based on voting-based TTS strategies often suffers from reward hacking problems. To address these issues, we propose DistriTTRL, which leverages the distribution prior of the model's confidence during RL to progressively optimize the reward signal, rather than relying solely on single-query rollouts. Additionally, we mitigate the phenomenon of consistent reward hacking caused by the voting-based TTS strategies through diversity-targeted penalties. Benefiting from this training mechanism where model capability and self-reward signals complement each other, and the mitigation of reward hacking, DistriTTRL has achieved significant performance improvements across multiple models and benchmarks.