# Distilling Self-Consistency into Verbal Confidence: A Pre-Registered Negative Result and Post-Hoc Rescue on Gemma 3 4B

arXiv: https://arxiv.org/abs/2604.24070 · HTML: https://mlsanity.com/paper/2604.24070/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Placed in Test-time compute and efficient thinking, one of 17 directions in LLM reasoning.
- Builds on 12 of its 18 references that are in the map.
- First posted on arXiv 27 Apr 2026.

## Placed in

- LLM reasoning / [Test-time compute and efficient thinking](https://mlsanity.com/lr/test-time-compute-and-efficient-thinking/)

## Builds on (most cited first, 12 of 12)

- [Measuring Massive Multitask Language Understanding](https://mlsanity.com/paper/2009.03300/) (2020)
- [TriviaQA: A Large Scale Distantly Supervised Challenge Dataset for Reading Comprehension](https://mlsanity.com/paper/1705.03551/) (2017)
- [Can LLMs Express Their Uncertainty? An Empirical Evaluation of Confidence Elicitation in LLMs](https://mlsanity.com/paper/2306.13063/) (2023)
- [Just Ask for Calibration: Strategies for Eliciting Calibrated Confidence Scores from Language Models Fine-Tuned with Human Feedback](https://mlsanity.com/paper/2305.14975/) (2023)
- [How do LLMs Compute Verbal Confidence](https://mlsanity.com/paper/2603.17839/) (2026)
- [Calibrating Verbalized Confidence with Self-Generated Distractors](https://mlsanity.com/paper/2509.25532/) (2025)
- [ADVICE: Answer-Dependent Verbalized Confidence Estimation](https://mlsanity.com/paper/2510.10913/) (2025)
- [Closing the Confidence-Faithfulness Gap in Large Language Models](https://mlsanity.com/paper/2603.25052/) (2026)
- [Wired for Overconfidence: A Mechanistic Perspective on Inflated Verbalized Confidence in LLMs](https://mlsanity.com/paper/2604.01457/) (2026)
- [Verbal Confidence Saturation in 3-9B Open-Weight Instruction-Tuned LLMs: A Pre-Registered Psychometric Validity Screen](https://mlsanity.com/paper/2604.22215/) (2026)
- [Quantisation Reshapes the Metacognitive Geometry of Language Models](https://mlsanity.com/paper/2604.08976/) (2026)
- [Cross-Entropy Is Load-Bearing: A Pre-Registered Scope Test of the K-Way Energy Probe on Bidirectional Predictive Coding](https://mlsanity.com/paper/2604.21286/) (2026)

## Abstract (arXiv)

Small instruct-tuned LLMs produce degenerate verbal confidence under minimal elicitation: ceiling rates above 95%, near-chance Type-2 AUROC, and Invalid validity profiles. We test whether confidence-conditioned supervised fine-tuning (CSFT) with self-consistency-derived targets can close the gap between internal information and verbal readout. A pre-registered Phase 0 protocol on Gemma 3 4B-it with a modal filter restricting training to items with correct modal answers produced a negative result: AUROC2 dropped from 0.554 to 0.509 due to label-entropy collapse in the training targets. An exploratory rescue removed the filter, training on all 2,000 calibration items. This produced a binary verbal correctness discriminator with AUROC2 = 0.774 on held-out TriviaQA, compressing a 10-sample self-consistency signal (AUROC2 = 0.999) into a single-pass readout exceeding logit entropy (0.701). The shuffled-target control showed no improvement (0.501). On MMLU, accuracy improved from 54.2% to 77.4% with the shuffled model at baseline (56.1%), supporting a target-dependent interpretation. The result is exploratory, binary rather than continuously calibrated, and observed at a single scale. It identifies two design lessons: confidence training requires label entropy, and correct targets regularise output format.