mlsanityData to 7 Oct 2026

From the Inside Out: Progressive Distribution Refinement for Confidence Calibration

arXiv 2603.16500 ↗First posted 17 Mar 2026cs.LG cs.CLcited by 0

On the map: LLM alignment / RLHF / preference optimisation → Reward hacking and misspecification

Abstract (arXiv)

Leveraging the model's internal information as the self-reward signal in Reinforcement Learning (RL) has received extensive attention due to its label-free nature. While prior works have made significant progress in applying the Test-Time Scaling (TTS) strategies to RL, the discrepancy in internal information between test and training remains inadequately addressed. Moreover, Test-Time Training based on voting-based TTS strategies often suffers from reward hacking problems. To address these issues, we propose DistriTTRL, which leverages the distribution prior of the model's confidence during RL to progressively optimize the reward signal, rather than relying solely on single-query rollouts. Additionally, we mitigate the phenomenon of consistent reward hacking caused by the voting-based TTS strategies through diversity-targeted penalties. Benefiting from this training mechanism where model capability and self-reward signals complement each other, and the mitigation of reward hacking, DistriTTRL has achieved significant performance improvements across multiple models and benchmarks.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Placed in Reward hacking and misspecification, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 24 of its 32 references that are in the map. First posted on arXiv 17 Mar 2026.

Builds on

24 of 32 references.

References · 24

  1. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,302
  2. 2025Qwen3 Technical Reportcited by 8,195
  3. 2022Self-Consistency Improves Chain of Thought Reasoning in Language Modelscited by 5,601
  4. 2021Measuring Mathematical Problem Solving With the MATH Datasetcited by 5,203
  5. 2023Let's Verify Step by Stepcited by 3,598
  6. 2023GPQA: A Graduate-Level Google-Proof Q&A Benchmarkcited by 2,748
  7. 2025Does Reinforcement Learning Really Incentivize Reasoning Capacity in LLMs Beyond the Base Model?cited by 948
  8. 2020Self-supervised Learning: Generative or Contrastivecited by 693
  9. 2024Titans: Learning to Memorize at Test Timecited by 303
  10. 2017"Zero-Shot" Super-Resolution using Deep Internal Learningcited by 285
  11. 2025MathArena: Evaluating LLMs on Uncontaminated Math Competitionscited by 275
  12. 2023A Survey of Confidence Estimation and Calibration in Large Language Modelscited by 253
12 more
  1. 2025TTRL: Test-Time Reinforcement Learningcited by 253
  2. 2025Deep Think with Confidencecited by 199
  3. 2025Scalable Best-of-N Selection for Large Language Models via Self-Certaintycited by 182
  4. 2020Semantic Photo Manipulation with a Generative Image Priorcited by 178
  5. 2024Fact-Checking the Output of Large Language Models via Token-Level Uncertainty Quantificationcited by 159
  6. 2025Right Question is Already Half the Answer: Fully Unsupervised LLM Reasoning Incentivizationcited by 124
  7. 2024The Surprising Effectiveness of Test-Time Training for Few-Shot Learningcited by 88
  8. 2024When to Trust LLMs: Aligning Confidence with Response Qualitycited by 42
  9. 2025Generalist Reward Models: Found Inside Large Language Modelscited by 34
  10. 2025Beyond Majority Voting: Towards Fine-grained and More Reliable Reward Signal for Test-Time Reinforcement Learningcited by 11
  11. 2025Majority of the Bests: Improving Best-of-N via Bootstrappingcited by 6
  12. 2026Believe Your Model: Distribution-Guided Confidence Calibrationcited by 2

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2026Countdown-Code: A Testbed for Studying The Emergence and Generalization of Reward Hacking in RLVR
  2. 2025Scaling Test-Time Compute Without Verification or RL is Suboptimal
  3. 2025Rewarding the Journey, Not Just the Destination: A Composite Path and Answer Self-Scoring Reward Mechanism for Test-Time Reinforcement Learning
  4. 2026Scaling with Confidence: Calibrating Confidence of LLMs for Adaptive Test Time Scaling
  5. 2026Multimodal Reward Hacking in Reinforcement Learning
  6. 2025AIRL-S: Unifying Reinforcement Learning and Search-Based Test-Time Scaling via Adversarial Inverse Reinforcement Learning
  7. 2026Tool Verification for Test-Time Reinforcement Learning
  8. 2026On the effectiveness of reward functions in reinforcement learning for confidence calibration of large language models