mlsanityData to 7 Oct 2026

Distilling Self-Consistency into Verbal Confidence: A Pre-Registered Negative Result and Post-Hoc Rescue on Gemma 3 4B

arXiv 2604.24070 ↗First posted 27 Apr 2026cs.CL cs.AIcited by 0

On the map: LLM reasoning → Test-time compute and efficient thinking

Abstract (arXiv)

Small instruct-tuned LLMs produce degenerate verbal confidence under minimal elicitation: ceiling rates above 95%, near-chance Type-2 AUROC, and Invalid validity profiles. We test whether confidence-conditioned supervised fine-tuning (CSFT) with self-consistency-derived targets can close the gap between internal information and verbal readout. A pre-registered Phase 0 protocol on Gemma 3 4B-it with a modal filter restricting training to items with correct modal answers produced a negative result: AUROC2 dropped from 0.554 to 0.509 due to label-entropy collapse in the training targets. An exploratory rescue removed the filter, training on all 2,000 calibration items. This produced a binary verbal correctness discriminator with AUROC2 = 0.774 on held-out TriviaQA, compressing a 10-sample self-consistency signal (AUROC2 = 0.999) into a single-pass readout exceeding logit entropy (0.701). The shuffled-target control showed no improvement (0.501). On MMLU, accuracy improved from 54.2% to 77.4% with the shuffled model at baseline (56.1%), supporting a target-dependent interpretation. The result is exploratory, binary rather than continuously calibrated, and observed at a single scale. It identifies two design lessons: confidence training requires label entropy, and correct targets regularise output format.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Placed in Test-time compute and efficient thinking, one of 17 directions in LLM reasoning. Builds on 12 of its 18 references that are in the map. First posted on arXiv 27 Apr 2026.

Builds on

12 of 18 references.

References · 12

  1. 2020Measuring Massive Multitask Language Understandingcited by 7,768
  2. 2017TriviaQA: A Large Scale Distantly Supervised Challenge Dataset for Reading Comprehensioncited by 2,895
  3. 2023Can LLMs Express Their Uncertainty? An Empirical Evaluation of Confidence Elicitation in LLMscited by 946
  4. 2023Just Ask for Calibration: Strategies for Eliciting Calibrated Confidence Scores from Language Models Fine-Tuned with Human Feedbackcited by 788
  5. 2026How do LLMs Compute Verbal Confidencecited by 23
  6. 2025Calibrating Verbalized Confidence with Self-Generated Distractorscited by 14
  7. 2025ADVICE: Answer-Dependent Verbalized Confidence Estimationcited by 8
  8. 2026Closing the Confidence-Faithfulness Gap in Large Language Modelscited by 8
  9. 2026Wired for Overconfidence: A Mechanistic Perspective on Inflated Verbalized Confidence in LLMscited by 4
  10. 2026Verbal Confidence Saturation in 3-9B Open-Weight Instruction-Tuned LLMs: A Pre-Registered Psychometric Validity Screencited by 2
  11. 2026Quantisation Reshapes the Metacognitive Geometry of Language Modelscited by 1
  12. 2026Cross-Entropy Is Load-Bearing: A Pre-Registered Scope Test of the K-Way Energy Probe on Bidirectional Predictive Codingcited by 1

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2025Confidence Improves Self-Consistency in LLMs
  2. 2025Verbalized Confidence Triggers Self-Verification: Emergent Behavior Without Explicit Reasoning Supervision
  3. 2026Confident in a Confidence Score: Investigating the Sensitivity of Confidence Scores to Supervised Fine-Tuning
  4. 2026An Analysis of Training-Free Self-Reported Confidence in Language Models
  5. 2026LLMs Should Express Uncertainty Explicitly
  6. 2026When Words Fall Short: Iterative Synergy Between Verbalized Reasoning and Hidden Features for LLM Confidence Estimation
  7. 2024Confidence v.s. Critique: A Decomposition of Self-Correction Capability for LLMs
  8. 2026Rethinking Verbalized Confidence for LLM-as-a-Judge: A Compatibility Shift on Post-2025 Proprietary Models