# Concurrent Criterion Validation of a Validity Screen for LLM Confidence Signals via Selective Prediction

arXiv: https://arxiv.org/abs/2604.17716 · HTML: https://mlsanity.com/paper/2604.17716/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 0 of its 3 references that are in the map.
- First posted on arXiv 20 Apr 2026.

## Built on by (most cited first, 2 of 2)

- [Verbal Confidence Saturation in 3-9B Open-Weight Instruction-Tuned LLMs: A Pre-Registered Psychometric Validity Screen](https://mlsanity.com/paper/2604.22215/) (2026)
- [Domain-level metacognitive monitoring in frontier LLMs: A 33-model atlas](https://mlsanity.com/paper/2605.06673/) (2026)

## Abstract (arXiv)

The validity screen (Cacioli, 2026d, 2026e) classifies LLM confidence signals as Valid, Indeterminate, or Invalid. We test whether these classifications predict selective prediction performance. Twenty frontier LLMs from seven families were evaluated on 524 items across six cognitive tracks. Valid models show mean Type 2 AUROC = .624 (SD = .048). Invalid models show mean AUROC = .357 (SD = .231). Cohen's d = 2.81, p = .002. The tiers order monotonically: Invalid (.357) < Indeterminate (.554) < Valid (.624). Split-half cross-validation yields median d = 1.77, P(d > 0) = 1.0 across 1,000 splits. The three-tier classification accounts for 47% of the variance in AUROC. DeepSeek-R1 drops from 85.3% accuracy at full coverage to 11.3% at 10% coverage. The screen predicts the criterion. For selective prediction, the screen matters.