mlsanityData to 14 Oct 2026

Rescaling Confidence: What Scale Design Reveals About LLM Metacognition

arXiv 2603.09309 ↗First posted 10 Mar 2026cs.AIcited by 3

Not on the map. Why?

Abstract (arXiv)

Verbalized confidence, in which LLMs report a numerical certainty score, is widely used to estimate uncertainty in black-box settings, yet the confidence scale itself (typically 0--100) is rarely examined. We show that this design choice is not neutral. Across six LLMs and three datasets, verbalized confidence is heavily discretized, with more than 78% of responses concentrating on just three round-number values. To investigate this phenomenon, we systematically manipulate confidence scales along three dimensions: granularity, boundary placement, and range regularity, and evaluate metacognitive sensitivity using meta-d'. We find that a 0--20 scale consistently improves metacognitive efficiency over the standard 0--100 format, while boundary compression degrades performance and round-number preferences persist even under irregular ranges. These results demonstrate that confidence scale design directly affects the quality of verbalized uncertainty and should be treated as a first-class experimental variable in LLM evaluation.

Key facts. Cited by 3 papers in the map (data through 14 Oct 2026). Builds on 12 of its 32 references that are in the map. First posted on arXiv 10 Mar 2026.

Builds on

12 of 32 references.

References · 12

  1. 2021Training Verifiers to Solve Math Word Problemscited by 10,109
  2. 2025Qwen3 Technical Reportcited by 8,820
  3. 2020Measuring Massive Multitask Language Understandingcited by 7,963
  4. 2017On Calibration of Modern Neural Networkscited by 4,960
  5. 2021TruthfulQA: Measuring How Models Mimic Human Falsehoodscited by 2,975
  6. 2022Language Models (Mostly) Know What They Knowcited by 1,856
  7. 2021Calibrate Before Use: Improving Few-Shot Performance of Language Modelscited by 1,392
  8. 2023Can LLMs Express Their Uncertainty? An Empirical Evaluation of Confidence Elicitation in LLMscited by 974
  9. 2022Teaching Models to Express Their Uncertainty in Wordscited by 751
  10. 2019Measuring Calibration in Deep Learningcited by 446
  11. 2019Verified Uncertainty Calibrationcited by 309
  12. 2024Relying on the Unreliable: The Impact of Language Models' Reluctance to Express Uncertaintycited by 106

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 3.

3 papers.