# Large Language Model Psychometrics: A Systematic Review of Evaluation, Validation, and Enhancement

arXiv: https://arxiv.org/abs/2505.08245 · HTML: https://mlsanity.com/paper/2505.08245/

## Key facts

- Cited by 45 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 13 May 2025.

## Built on by (most cited first, 30 of 45)

- [Lost in Benchmarks? Rethinking Large Language Model Benchmarking with Item Response Theory](https://mlsanity.com/paper/2505.15055/) (2025)
- [Persistent Instability in LLM's Personality Measurements: Effects of Scale, Reasoning, and Conversation History](https://mlsanity.com/paper/2508.04826/) (2025)
- [The Benchmarking Epistemology: Validity Theory for Evaluating Machine Learning Models](https://mlsanity.com/paper/2510.23191/) (2025)
- [From Prompts to Constructs: A Dual-Validity Framework for Large Language Model Research in Psychology](https://mlsanity.com/paper/2506.16697/) (2025)
- [Do Psychometric Tests Work for Large Language Models? Evaluation of Tests on Sexism, Racism, and Morality](https://mlsanity.com/paper/2510.11254/) (2025)
- [A validity-guided workflow for robust large language model research in psychology](https://mlsanity.com/paper/2507.04491/) (2025)
- [Beyond the Mean: Within-Model Reliable Change Detection for LLM Evaluation](https://mlsanity.com/paper/2604.27405/) (2026)
- [PReMISE: Policy Rubrics as Measurement Specifications for LLM Judges](https://mlsanity.com/paper/2605.30803/) (2026)
- [An LLM-Native Psychometric Instrument Reveals a Self-Report--Behavior Gap Across 25 Models](https://mlsanity.com/paper/2606.09843/) (2026)
- [A Detailed Factor Analysis for the Political Compass Test: Navigating Ideologies of Large Language Models](https://mlsanity.com/paper/2506.22493/) (2025)
- [Psychometric Item Validation Using Virtual Respondents with Trait-Response Mediators](https://mlsanity.com/paper/2507.05890/) (2025)
- [Scaling Law in LLM Simulated Personality: More Detailed and Realistic Persona Profile Is All You Need](https://mlsanity.com/paper/2510.11734/) (2025)
- [AI-exhibited Personality Traits Can Shape Human Self-concept through Conversations](https://mlsanity.com/paper/2601.12727/) (2026)
- [GenPT: Beyond Self-Report for Reliable LLM Psychometrics via Generative Projective Testing](https://mlsanity.com/paper/2606.00860/) (2026)
- [Beyond the Mean: Three-Axis Fidelity for Aligning LLM-Based Survey Simulators from Small Pilot Data](https://mlsanity.com/paper/2606.28963/) (2026)
- [A Hierarchical Imprecise Probability Approach to Reliability Assessment of Large Language Models](https://mlsanity.com/paper/2511.00527/) (2025)
- [Quantifying and Mitigating Socially Desirable Responding in LLMs: A Desirability-Matched Graded Forced-Choice Psychometric Study](https://mlsanity.com/paper/2602.17262/) (2026)
- [Context-Value-Action Architecture for Value-Driven Large Language Model Agents](https://mlsanity.com/paper/2604.05939/) (2026)
- [Before You Interpret the Profile: Validity Scaling for LLM Metacognitive Self-Report](https://mlsanity.com/paper/2604.17707/) (2026)
- [Agent-ValueBench: A Comprehensive Benchmark for Evaluating Agent Values](https://mlsanity.com/paper/2605.10365/) (2026)
- [Apparent Psychological Profiles of Large Language Models are Largely a Measurement Artifact](https://mlsanity.com/paper/2606.20205/) (2026)
- [Implicit Values Embedded in How Humans and LLMs Complete Subjective Everyday Tasks](https://mlsanity.com/paper/2510.03384/) (2025)
- [Simulated Self-Assessment in Large Language Models: A Psychometric Approach to AI Self-Efficacy](https://mlsanity.com/paper/2511.19872/) (2025)
- [QSTN: A Modular Framework for Robust Questionnaire Inference with Large Language Models](https://mlsanity.com/paper/2512.08646/) (2025)
- [PieArena: Ranking and Profiling Language Agents in Realistic Negotiation Scenarios](https://mlsanity.com/paper/2602.05302/) (2026)
- [Value Alignment Tax: Measuring Value Trade-offs in LLM Alignment](https://mlsanity.com/paper/2602.12134/) (2026)
- [BenchBrowser: Retrieving Evidence for Evaluating Benchmark Validity](https://mlsanity.com/paper/2603.18019/) (2026)
- [Rethinking Psychometric Evaluation of LLMs: When and Why Self-Reports Predict Behavior](https://mlsanity.com/paper/2606.12730/) (2026)
- [Your LLM, Your Style: Behavioral Mode Axes for LLM Behavioral Control](https://mlsanity.com/paper/2608.10703/) (2026)
- [Beyond Asking: A Pipeline for Personalized Game Generation that Reads Players from Behavior](https://mlsanity.com/paper/2608.16196/) (2026)

## Abstract (arXiv)

The advancement of large language models (LLMs) has outpaced traditional evaluation methodologies. This progress presents novel challenges, such as measuring human-like psychological constructs, moving beyond static and task-specific benchmarks, and establishing human-centered evaluation. These challenges intersect with psychometrics, the science of quantifying the intangible aspects of human psychology, such as personality, values, and intelligence. This review paper introduces and synthesizes the emerging interdisciplinary field of LLM Psychometrics, which leverages psychometric instruments, theories, and principles to evaluate, understand, and enhance LLMs. The reviewed literature systematically shapes benchmarking principles, broadens evaluation scopes, refines methodologies, validates results, and advances LLM capabilities. Diverse perspectives are integrated to provide a structured framework for researchers across disciplines, enabling a more comprehensive understanding of this nascent field. Ultimately, the review provides actionable insights for developing future evaluation paradigms that align with human-level AI and promote the advancement of human-centered AI systems for societal benefit. A curated repository of LLM psychometric resources is available at https://github.com/valuebyte-ai/Awesome-LLM-Psychometrics.