Rescaling Confidence: What Scale Design Reveals About LLM Metacognition
Not on the map. Why?
Abstract (arXiv)
Verbalized confidence, in which LLMs report a numerical certainty score, is widely used to estimate uncertainty in black-box settings, yet the confidence scale itself (typically 0--100) is rarely examined. We show that this design choice is not neutral. Across six LLMs and three datasets, verbalized confidence is heavily discretized, with more than 78% of responses concentrating on just three round-number values. To investigate this phenomenon, we systematically manipulate confidence scales along three dimensions: granularity, boundary placement, and range regularity, and evaluate metacognitive sensitivity using meta-d'. We find that a 0--20 scale consistently improves metacognitive efficiency over the standard 0--100 format, while boundary compression degrades performance and round-number preferences persist even under irregular ranges. These results demonstrate that confidence scale design directly affects the quality of verbalized uncertainty and should be treated as a first-class experimental variable in LLM evaluation.
Key facts. Cited by 3 papers in the map (data through 14 Oct 2026). Builds on 12 of its 32 references that are in the map. First posted on arXiv 10 Mar 2026.
Builds on
12 of 32 references.
References · 12
- 2021Training Verifiers to Solve Math Word Problemscited by 10,109
- 2025Qwen3 Technical Reportcited by 8,820
- 2020Measuring Massive Multitask Language Understandingcited by 7,963
- 2017On Calibration of Modern Neural Networkscited by 4,960
- 2021TruthfulQA: Measuring How Models Mimic Human Falsehoodscited by 2,975
- 2022Language Models (Mostly) Know What They Knowcited by 1,856
- 2021Calibrate Before Use: Improving Few-Shot Performance of Language Modelscited by 1,392
- 2023Can LLMs Express Their Uncertainty? An Empirical Evaluation of Confidence Elicitation in LLMscited by 974
- 2022Teaching Models to Express Their Uncertainty in Wordscited by 751
- 2019Measuring Calibration in Deep Learningcited by 446
- 2019Verified Uncertainty Calibrationcited by 309
- 2024Relying on the Unreliable: The Impact of Language Models' Reluctance to Express Uncertaintycited by 106
Built on by
Citing papers per year: 2026: 3.
3 papers.
- 2026I-CALM: Incentivizing Confidence-Aware Abstention for LLM Selective Answeringcited by 5
- 2026Beyond Confidence: Rethinking Self-Assessments for Performance Prediction in LLMscited by 1
- 2026Rethinking Verbalized Confidence for LLM-as-a-Judge: A Compatibility Shift on Post-2025 Proprietary Modelscited by 1
- 2026Rethinking Verbalized Confidence for LLM-as-a-Judge: A Compatibility Shift on Post-2025 Proprietary Models10 Sep 2026
- 2026Beyond Confidence: Rethinking Self-Assessments for Performance Prediction in LLMs8 May 2026
- 2026I-CALM: Incentivizing Confidence-Aware Abstention for LLM Selective Answering5 Apr 2026