# Do Psychometric Tests Work for Large Language Models? Evaluation of Tests on Sexism, Racism, and Morality

arXiv: https://arxiv.org/abs/2510.11254 · HTML: https://mlsanity.com/paper/2510.11254/

## Key facts

- Cited by 9 papers in the map (data through 14 Oct 2026).
- Builds on 22 of its 112 references that are in the map.
- First posted on arXiv 13 Oct 2025.

## Builds on (most cited first, 22 of 22)

- [The Llama 3 Herd of Models](https://mlsanity.com/paper/2407.21783/) (2024)
- [Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks](https://mlsanity.com/paper/1908.10084/) (2019)
- [LLM Evaluators Recognize and Favor Their Own Generations](https://mlsanity.com/paper/2404.13076/) (2024)
- [Evaluating the Moral Beliefs Encoded in LLMs](https://mlsanity.com/paper/2307.14324/) (2023)
- ["Kelly is a Warm Person, Joseph is a Role Model": Gender Biases in LLM-Generated Reference Letters](https://mlsanity.com/paper/2310.09219/) (2023)
- [Political Compass or Spinning Arrow? Towards More Meaningful Evaluations for Values and Opinions in Large Language Models](https://mlsanity.com/paper/2402.16786/) (2024)
- [Who is GPT-3? An Exploration of Personality, Values and Demographics](https://mlsanity.com/paper/2209.14338/) (2022)
- [Moral Foundations of Large Language Models](https://mlsanity.com/paper/2310.15337/) (2023)
- [DailyDilemmas: Revealing Value Preferences of LLMs with Quandaries of Daily Life](https://mlsanity.com/paper/2410.02683/) (2024)
- [How Well Do LLMs Represent Values Across Cultures? Empirical Analysis of LLM Responses Based on Hofstede Cultural Dimensions](https://mlsanity.com/paper/2406.14805/) (2024)
- [Mind the Value-Action Gap: Do LLMs Act in Alignment with Their Values?](https://mlsanity.com/paper/2501.15463/) (2025)
- [You don't need a personality test to know these models are unreliable: Assessing the Reliability of Large Language Models on Psychometric Instruments](https://mlsanity.com/paper/2311.09718/) (2023)
- [Large Language Model Psychometrics: A Systematic Review of Evaluation, Validation, and Enhancement](https://mlsanity.com/paper/2505.08245/) (2025)
- [Exploring the psychology of LLMs' Moral and Legal Reasoning](https://mlsanity.com/paper/2308.01264/) (2023)
- [Centaur: a foundation model of human cognition](https://mlsanity.com/paper/2410.20268/) (2024)
- [The Personality Illusion: Revealing Dissociation Between Self-Reports & Behavior in LLMs](https://mlsanity.com/paper/2509.03730/) (2025)
- [Actions Speak Louder than Words: Agent Decisions Reveal Implicit Biases in Language Models](https://mlsanity.com/paper/2501.17420/) (2025)
- [AnthroScore: A Computational Linguistic Measure of Anthropomorphism](https://mlsanity.com/paper/2402.02056/) (2024)
- [Denevil: Towards Deciphering and Navigating the Ethical Values of Large Language Models via Instruction Learning](https://mlsanity.com/paper/2310.11053/) (2023)
- [Are Large Language Models Moral Hypocrites? A Study Based on Moral Foundations](https://mlsanity.com/paper/2405.11100/) (2024)
- [Inducing anxiety in large language models can induce bias](https://mlsanity.com/paper/2304.11111/) (2023)
- [Position: Stop Evaluating AI with Human Tests, Develop Principled, AI-specific Tests instead](https://mlsanity.com/paper/2507.23009/) (2025)

## Built on by (most cited first, 9 of 9)

- [Different Demographic Cues Yield Inconsistent Conclusions About LLM Personalization and Bias](https://mlsanity.com/paper/2601.18486/) (2026)
- [Every Token Counts: Exact Likert-Scale Distributions for Measuring LLM Attitudes and Biases](https://mlsanity.com/paper/2608.10503/) (2026)
- [Evaluating Cognitive Age Alignment in Interactive AI Agents](https://mlsanity.com/paper/2605.17894/) (2026)
- [Apparent Psychological Profiles of Large Language Models are Largely a Measurement Artifact](https://mlsanity.com/paper/2606.20205/) (2026)
- [Persona-Model Collapse in Emergent Misalignment](https://mlsanity.com/paper/2605.12850/) (2026)
- [Rethinking Psychometric Evaluation of LLMs: When and Why Self-Reports Predict Behavior](https://mlsanity.com/paper/2606.12730/) (2026)
- [AuAu: A Benchmark for Auditing Authoritarian Alignment in Large Language Models](https://mlsanity.com/paper/2606.16127/) (2026)
- [CogArena: A Multimethod Evaluation of Cognitive Ability Structure in Large Language Models](https://mlsanity.com/paper/2607.24999/) (2026)
- [When benchmark inferences do not compose: Projectibility in AI evaluation](https://mlsanity.com/paper/2607.26159/) (2026)

## Abstract (arXiv)

Psychometric tests are increasingly used to assess psychological constructs in large language models (LLMs). However, it remains unclear whether these tests -- originally developed for humans -- yield meaningful results when applied to LLMs. In this study, we systematically evaluate the reliability and validity of human psychometric tests on 17 LLMs for three constructs: sexism, racism, and morality. We find moderate reliability across multiple item and prompt variations. Validity is evaluated through both convergent (i.e., testing theory-based inter-test correlations) and ecological approaches (i.e., testing the alignment between tests scores and behavior in real-world downstream tasks). Crucially, we find that psychometric test scores do not align, and in some cases even negatively correlate with, model behavior in downstream tasks, indicating low ecological validity. Our results highlight that systematic evaluations of psychometric tests on LLMs are essential before interpreting their scores. Our findings also suggest that psychometric tests designed for humans cannot be applied directly to LLMs without adaptation.