# Verbal Confidence Saturation in 3-9B Open-Weight Instruction-Tuned LLMs: A Pre-Registered Psychometric Validity Screen

arXiv: https://arxiv.org/abs/2604.22215 · HTML: https://mlsanity.com/paper/2604.22215/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 10 of its 21 references that are in the map.
- First posted on arXiv 24 Apr 2026.

## Builds on (most cited first, 10 of 10)

- [TriviaQA: A Large Scale Distantly Supervised Challenge Dataset for Reading Comprehension](https://mlsanity.com/paper/1705.03551/) (2017)
- [A Survey of Confidence Estimation and Calibration in Large Language Models](https://mlsanity.com/paper/2311.08298/) (2023)
- [How do LLMs Compute Verbal Confidence](https://mlsanity.com/paper/2603.17839/) (2026)
- [Metacognition and Uncertainty Communication in Humans and Large Language Models](https://mlsanity.com/paper/2504.14045/) (2025)
- [Calibrating Verbalized Confidence with Self-Generated Distractors](https://mlsanity.com/paper/2509.25532/) (2025)
- [ADVICE: Answer-Dependent Verbalized Confidence Estimation](https://mlsanity.com/paper/2510.10913/) (2025)
- [The Metacognitive Monitoring Battery: A Cross-Domain Benchmark for LLM Self-Monitoring](https://mlsanity.com/paper/2604.15702/) (2026)
- [Before You Interpret the Profile: Validity Scaling for LLM Metacognitive Self-Report](https://mlsanity.com/paper/2604.17707/) (2026)
- [Screen Before You Interpret: A Portable Validity Protocol for Benchmark-Based LLM Confidence Signals](https://mlsanity.com/paper/2604.17714/) (2026)
- [Concurrent Criterion Validation of a Validity Screen for LLM Confidence Signals via Selective Prediction](https://mlsanity.com/paper/2604.17716/) (2026)

## Built on by (most cited first, 2 of 2)

- [Distilling Self-Consistency into Verbal Confidence: A Pre-Registered Negative Result and Post-Hoc Rescue on Gemma 3 4B](https://mlsanity.com/paper/2604.24070/) (2026)
- [On the Pitfalls of Verbalized Confidence Priors for Calibrating Large Reasoning Models](https://mlsanity.com/paper/2609.32470/) (2026)

## Abstract (arXiv)

Verbal confidence elicitation is widely used to extract uncertainty estimates from LLMs. We tested whether seven instruction-tuned open-weight models (3-9B parameters, four families) produce verbalised confidence that meets minimal validity criteria for item-level Type-2 discrimination under minimal numeric elicitation with greedy decoding. In a pre-registered study (OSF: osf.io/azbvx), 524 TriviaQA items were administered under numeric (0-100) and categorical (10-class) elicitation to eight models at Q5_K_M quantisation on consumer hardware, yielding 8,384 deterministic trials. A psychometric validity screen was applied to each model-format cell. All seven instruct models were classified Invalid on numeric confidence (H2 confirmed, 7/7 vs. predicted >=4/7), with a mean ceiling rate of 91.7% (H1 confirmed). Categorical elicitation did not rescue validity. Instead, it disrupted task performance in six of seven models, producing accuracy below 5% (H4 not confirmed). Token-level logprobability did not usefully predict verbalised confidence under the observed variance regime (H5 confirmed, mean cross-validated R^2 < 0.01). Within the reasoning-distilled model, reasoning-trace length showed a strong negative partial correlation with confidence (rho = -0.36, p < .001), consistent with the Reasoning Contamination Effect. These results do not imply that internal uncertainty representations are absent. They show that minimal verbal elicitation fails to preserve internal signals at the output interface in this model-size regime. Psychometric screening should precede any downstream use of such signals.