# Measure what Matters: Psychometric Evaluation of AI with Situational Judgment Tests

arXiv: https://arxiv.org/abs/2510.22170 · HTML: https://mlsanity.com/paper/2510.22170/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 21 of its 61 references that are in the map.
- First posted on arXiv 25 Oct 2025.

## Builds on (most cited first, 21 of 21)

- [Efficient Memory Management for Large Language Model Serving with PagedAttention](https://mlsanity.com/paper/2309.06180/) (2023)
- [Generative Agents: Interactive Simulacra of Human Behavior](https://mlsanity.com/paper/2304.03442/) (2023)
- [LLM Agents Grounded in Self-Reports Enable General-Purpose Simulation of Individuals](https://mlsanity.com/paper/2411.10109/) (2024)
- [PersonaLLM: Investigating the Ability of Large Language Models to Express Personality Traits](https://mlsanity.com/paper/2305.02547/) (2023)
- [Large Language Model as Attributed Training Data Generator: A Tale of Diversity and Bias](https://mlsanity.com/paper/2306.15895/) (2023)
- [Personality Traits in Large Language Models](https://mlsanity.com/paper/2307.00184/) (2023)
- [Quantifying the Persona Effect in LLM Simulations](https://mlsanity.com/paper/2402.10811/) (2024)
- [Do LLMs Have Distinct and Consistent Personality? TRAIT: Personality Testset designed for LLMs with Psychometrics](https://mlsanity.com/paper/2406.14703/) (2024)
- [Standardizing the Measurement of Text Diversity: A Tool and a Comparative Analysis of Scores](https://mlsanity.com/paper/2403.00553/) (2024)
- [Evaluating Cultural Adaptability of a Large Language Model via Simulation of Synthetic Personas](https://mlsanity.com/paper/2408.06929/) (2024)
- [Leveraging LLM-Respondents for Item Evaluation: a Psychometric Analysis](https://mlsanity.com/paper/2407.10899/) (2024)
- ['Simulacrum of Stories': Examining Large Language Models as Qualitative Research Participants](https://mlsanity.com/paper/2409.19430/) (2024)
- [Large Language Models Do Not Simulate Human Psychology](https://mlsanity.com/paper/2508.06950/) (2025)
- [Algorithmic Fidelity of Large Language Models in Generating Synthetic German Public Opinions: A Case Study](https://mlsanity.com/paper/2412.13169/) (2024)
- [Framework-Based Qualitative Analysis of Free Responses of Large Language Models: Algorithmic Fidelity](https://mlsanity.com/paper/2309.06364/) (2023)
- [From Lab to Field: Real-World Evaluation of an AI-Driven Smart Video Solution to Enhance Community Safety](https://mlsanity.com/paper/2312.02078/) (2023)
- [Generative Personality Simulation via Theory-Informed Structured Interview](https://mlsanity.com/paper/2502.12109/) (2025)
- [Deterministic AI Agent Personality Expression through Standard Psychological Diagnostics](https://mlsanity.com/paper/2503.17085/) (2025)
- [Using LLMs to identify features of personal and professional skills in an open-response situational judgment test](https://mlsanity.com/paper/2507.13881/) (2025)
- [Assessing the nature of large language models: A caution against anthropocentrism](https://mlsanity.com/paper/2309.07683/) (2023)
- [AI-driven formative assessment and adaptive learning in data-science education: Evaluating an LLM-powered virtual teaching assistant](https://mlsanity.com/paper/2509.20369/) (2025)

## Abstract (arXiv)

Persona conditioning is widely used to steer large language model (LLM) behavior, but it is unclear whether it induces stable behavioral structure or superficial variation. We propose a framework to measure consistent behavioral tendencies using situational judgment tests (SJTs), multidimensional item response theory (MIRT), and structured synthetic personas, treating responses as observations of latent behavioral variables. Across large-scale SJT and persona datasets, we find that persona-conditioned behaviors are stable across runs, latent trait scores predict external benchmarks (e.g., TruthfulQA, EmoBench), and MIRT reveals consistent latent structure. We validate these results through human annotation, benchmark evaluation, and internal consistency analyses. We interpret these traits not as human personality, but as stable behavioral tendencies expressed across contexts. Our results show that scenario-based psychometric evaluation provides a more reliable alternative to classical self-report approaches for assessing LLM behavior, and we release datasets to support further study.