# An LLM-Native Psychometric Instrument Reveals a Self-Report--Behavior Gap Across 25 Models

arXiv: https://arxiv.org/abs/2606.09843 · HTML: https://mlsanity.com/paper/2606.09843/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- Builds on 13 of its 44 references that are in the map.
- First posted on arXiv 24 Apr 2026.

## Builds on (most cited first, 13 of 13)

- [Constitutional AI: Harmlessness from AI Feedback](https://mlsanity.com/paper/2212.08073/) (2022)
- [A General Language Assistant as a Laboratory for Alignment](https://mlsanity.com/paper/2112.00861/) (2021)
- [Towards Understanding Sycophancy in Language Models](https://mlsanity.com/paper/2310.13548/) (2023)
- [Length-Controlled AlpacaEval: A Simple Way to Debias Automatic Evaluators](https://mlsanity.com/paper/2404.04475/) (2024)
- [Discovering Language Model Behaviors with Model-Written Evaluations](https://mlsanity.com/paper/2212.09251/) (2022)
- [Judging the Judges: A Systematic Study of Position Bias in LLM-as-a-Judge](https://mlsanity.com/paper/2406.07791/) (2024)
- [Self-Preference Bias in LLM-as-a-Judge](https://mlsanity.com/paper/2410.21819/) (2024)
- [Art or Artifice? Large Language Models and the False Promise of Creativity](https://mlsanity.com/paper/2309.14556/) (2023)
- [Large Language Model Psychometrics: A Systematic Review of Evaluation, Validation, and Enhancement](https://mlsanity.com/paper/2505.08245/) (2025)
- [Evaluating Personality Traits in Large Language Models: Insights from Psychological Questionnaires](https://mlsanity.com/paper/2502.05248/) (2025)
- [Evaluating Large Language Models with Psychometrics](https://mlsanity.com/paper/2406.17675/) (2024)
- [On Calibration of Large Language Models: From Response To Capability](https://mlsanity.com/paper/2602.13540/) (2026)
- [Evaluating and Mitigating LLM-as-a-judge Bias in Communication Systems](https://mlsanity.com/paper/2510.12462/) (2025)

## Built on by (most cited first, 4 of 4)

- [The Two-Process Theory of Machine Self-Report](https://mlsanity.com/paper/2607.20082/) (2026)
- [What the Window Does Not Contain: Auditing Provenance in a Document-Grounded Instability Benchmark](https://mlsanity.com/paper/2609.06147/) (2026)
- [Off-Target Effects of Response-Style Alignment in a Korean 27B Language Model](https://mlsanity.com/paper/2609.11291/) (2026)
- [On the Behavioral Traits of LLM Agents](https://mlsanity.com/paper/2609.32776/) (2026)

## Abstract (arXiv)

Do large language models' (LLMs') answers to self-report questionnaires predict how they behave? Prior work finds they do not, but it uses human personality inventories, so the gap could reflect borrowed human constructs rather than LLM self-report itself. We test this with a self-report instrument built from LLM-specific behaviors (e.g., over-refusal, unsolicited disclaimers) whose structure is derived bottom-up. Administering 300 items 30 times to 25 LLMs from 17 developers yields five replicable, reliable factors (Tucker φ ≥ .957, α ≥ .930). We compare these self-reports with 2,500 open-ended behavioral samples rated by 151 humans and an LLM-judge ensemble. Humans and judges agree about model behavior (barr = .51), but self-report barely tracks human ratings (barr = .09, 95% CI [-.07, .18]) or rater-free text measures, and correcting for criterion unreliability leaves four of five factors near zero. Verbosity is the partial exception (r = .40, 71% of its reliability ceiling). On Responsiveness, self-report tracks LLM judges more than humans (r = .53 vs. .18; Steiger p = .04), and controlling for length and formatting does not remove this: agreement between LLM judges and LLM self-report is weak evidence that either tracks human judgment.