# Before You Interpret the Profile: Validity Scaling for LLM Metacognitive Self-Report

arXiv: https://arxiv.org/abs/2604.17707 · HTML: https://mlsanity.com/paper/2604.17707/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 6 of its 16 references that are in the map.
- First posted on arXiv 20 Apr 2026.

## Builds on (most cited first, 6 of 6)

- [Language Models (Mostly) Know What They Know](https://mlsanity.com/paper/2207.05221/) (2022)
- [Large Language Model Psychometrics: A Systematic Review of Evaluation, Validation, and Enhancement](https://mlsanity.com/paper/2505.08245/) (2025)
- [BAS: A Decision-Theoretic Approach to Evaluating Large Language Model Confidence](https://mlsanity.com/paper/2604.03216/) (2026)
- [Metacognitive Myopia in Large Language Models](https://mlsanity.com/paper/2408.05568/) (2024)
- [Do LLMs Know What They Know? Measuring Metacognitive Efficiency with Signal Detection Theory](https://mlsanity.com/paper/2603.25112/) (2026)
- [The Metacognitive Monitoring Battery: A Cross-Domain Benchmark for LLM Self-Monitoring](https://mlsanity.com/paper/2604.15702/) (2026)

## Built on by (most cited first, 2 of 2)

- [Verbal Confidence Saturation in 3-9B Open-Weight Instruction-Tuned LLMs: A Pre-Registered Psychometric Validity Screen](https://mlsanity.com/paper/2604.22215/) (2026)
- [Domain-level metacognitive monitoring in frontier LLMs: A 33-model atlas](https://mlsanity.com/paper/2605.06673/) (2026)

## Abstract (arXiv)

Clinical personality assessment screens response validity before interpreting substantive scales. LLM evaluation does not. We apply the validity scaling framework from the PAI and MMPI-3 to metacognitive probe data from 20 frontier models across 524 items. Six validity indices are operationalised: L (maintaining confidence on errors), K (betting on errors), F (withdrawing consensus-endorsed items), Fp (withdrawing correct answers), RBS (inverted monitoring), and TRIN (fixed responding). A tiered classification system identifies four models as construct-level invalid and two as elevated. Valid-profile models produce item-sensitive confidence (mean r = .18, 14 of 16 significant). Invalid-profile models do not (mean r = -.20, d = 2.17, p = .001). Chain-of-thought training produces two opposite response distortions. Two latent dimensions account for 94.6% of index variance. Companion papers extract a portable screening protocol (Cacioli, 2026e) and validate it against selective prediction (Cacioli, 2026f). All data and code: https://github.com/synthiumjp/validity-scaling-llm