mlsanityData to 13 Oct 2026

Before You Interpret the Profile: Validity Scaling for LLM Metacognitive Self-Report

arXiv 2604.17707 ↗First posted 20 Apr 2026cs.CL cs.AIcited by 2

Not on the map. Why?

Abstract (arXiv)

Clinical personality assessment screens response validity before interpreting substantive scales. LLM evaluation does not. We apply the validity scaling framework from the PAI and MMPI-3 to metacognitive probe data from 20 frontier models across 524 items. Six validity indices are operationalised: L (maintaining confidence on errors), K (betting on errors), F (withdrawing consensus-endorsed items), Fp (withdrawing correct answers), RBS (inverted monitoring), and TRIN (fixed responding). A tiered classification system identifies four models as construct-level invalid and two as elevated. Valid-profile models produce item-sensitive confidence (mean r = .18, 14 of 16 significant). Invalid-profile models do not (mean r = -.20, d = 2.17, p = .001). Chain-of-thought training produces two opposite response distortions. Two latent dimensions account for 94.6% of index variance. Companion papers extract a portable screening protocol (Cacioli, 2026e) and validate it against selective prediction (Cacioli, 2026f). All data and code: https://github.com/synthiumjp/validity-scaling-llm

Key facts. Cited by 2 papers in the map (data through 13 Oct 2026). Builds on 6 of its 16 references that are in the map. First posted on arXiv 20 Apr 2026.

Builds on

6 of 16 references.

References · 6

  1. 2022Language Models (Mostly) Know What They Knowcited by 1,823
  2. 2025Large Language Model Psychometrics: A Systematic Review of Evaluation, Validation, and Enhancementcited by 43
  3. 2024Metacognitive Myopia in Large Language Modelscited by 6
  4. 2026Do LLMs Know What They Know? Measuring Metacognitive Efficiency with Signal Detection Theorycited by 6
  5. 2026BAS: A Decision-Theoretic Approach to Evaluating Large Language Model Confidencecited by 6
  6. 2026The Metacognitive Monitoring Battery: A Cross-Domain Benchmark for LLM Self-Monitoringcited by 5

Built on by

Cited by year (2026 to 13 Oct)

Citing papers per year: 2026: 2.

2 papers.

  1. 2026Verbal Confidence Saturation in 3-9B Open-Weight Instruction-Tuned LLMs: A Pre-Registered Psychometric Validity Screencited by 2
  2. 2026Domain-level metacognitive monitoring in frontier LLMs: A 33-model atlascited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026Concurrent Criterion Validation of a Validity Screen for LLM Confidence Signals via Selective Prediction
  2. 2025In Silico Development of Psychometric Scales: Feasibility of Representative Population Data Simulation with LLMs
  3. 2025Can LLMs Infer Personality from Real World Conversations?
  4. 2026Evaluation Drift in LLM Personality Induction: Are We Moving the Goalpost?
  5. 2026An LLM-Native Psychometric Instrument Reveals a Self-Report--Behavior Gap Across 25 Models
  6. 2026Aligning LLM-Simulated and Human Examinees for Psychometric Calibration: A Cognitive Diagnostic Profiling Approach
  7. 2026LLMs Show No Signs Of Individuated Metacognition
  8. 2025ALIGNS: Unlocking nomological networks in psychological measurement through a large language model