# Evaluating General-Purpose AI with Psychometrics

arXiv: https://arxiv.org/abs/2310.16379 · HTML: https://mlsanity.com/paper/2310.16379/

## Key facts

- Cited by 23 papers in the map (data through 14 Oct 2026).
- Builds on 24 of its 119 references that are in the map.
- First posted on arXiv 25 Oct 2023.

## Builds on (most cited first, 24 of 24)

- [Chain-of-Thought Prompting Elicits Reasoning in Large Language Models](https://mlsanity.com/paper/2201.11903/) (2022)
- [On the Opportunities and Risks of Foundation Models](https://mlsanity.com/paper/2108.07258/) (2021)
- [Prefix-Tuning: Optimizing Continuous Prompts for Generation](https://mlsanity.com/paper/2101.00190/) (2021)
- [Sparks of Artificial General Intelligence: Early experiments with GPT-4](https://mlsanity.com/paper/2303.12712/) (2023)
- [Beyond the Imitation Game: Quantifying and extrapolating the capabilities of language models](https://mlsanity.com/paper/2206.04615/) (2022)
- [A Survey on Evaluation of Large Language Models](https://mlsanity.com/paper/2307.03109/) (2023)
- [How Can We Know What Language Models Know?](https://mlsanity.com/paper/1911.12543/) (2019)
- [CodeGen: An Open Large Language Model for Code with Multi-Turn Program Synthesis](https://mlsanity.com/paper/2203.13474/) (2022)
- [Using cognitive psychology to understand GPT-3](https://mlsanity.com/paper/2206.14576/) (2022)
- [HaluEval: A Large-Scale Hallucination Evaluation Benchmark for Large Language Models](https://mlsanity.com/paper/2305.11747/) (2023)
- [Role-Play with Large Language Models](https://mlsanity.com/paper/2305.16367/) (2023)
- [A Survey on Neural Network Interpretability](https://mlsanity.com/paper/2012.14261/) (2020)
- [Emergent Analogical Reasoning in Large Language Models](https://mlsanity.com/paper/2212.09196/) (2022)
- [Data Contamination: From Memorization to Exploitation](https://mlsanity.com/paper/2203.08242/) (2022)
- [Harms of Gender Exclusivity and Challenges in Non-Binary Representation in Language Technologies](https://mlsanity.com/paper/2108.12084/) (2021)
- [Towards a Science of Human-AI Decision Making: A Survey of Empirical Studies](https://mlsanity.com/paper/2112.11471/) (2021)
- [Moral Foundations of Large Language Models](https://mlsanity.com/paper/2310.15337/) (2023)
- [How Robust is GPT-3.5 to Predecessors? A Comprehensive Study on Language Understanding Tasks](https://mlsanity.com/paper/2303.00293/) (2023)
- [Emotional Intelligence of Large Language Models](https://mlsanity.com/paper/2307.09042/) (2023)
- [Evaluating Evaluation Metrics: A Framework for Analyzing NLG Evaluation Metrics using Measurement Theory](https://mlsanity.com/paper/2305.14889/) (2023)
- [Revealing the structure of language model capabilities](https://mlsanity.com/paper/2306.10062/) (2023)
- [GEMv2: Multilingual NLG Benchmarking in a Single Line of Code](https://mlsanity.com/paper/2206.11249/) (2022)
- [Evidence of interrelated cognitive-like capabilities in large language models: Indications of artificial general intelligence or achievement?](https://mlsanity.com/paper/2310.11616/) (2023)
- [Inferring Capabilities from Task Performance with Bayesian Triangulation](https://mlsanity.com/paper/2309.11975/) (2023)

## Built on by (most cited first, 23 of 23)

- [Large Language Model Agent: A Survey on Methodology, Applications and Challenges](https://mlsanity.com/paper/2503.21460/) (2025)
- [Evaluating Large Language Models with Psychometrics](https://mlsanity.com/paper/2406.17675/) (2024)
- [Comparison requires valid measurement: Rethinking attack success rate comparisons in AI red teaming](https://mlsanity.com/paper/2601.18076/) (2026)
- [The Benchmarking Epistemology: Validity Theory for Evaluating Machine Learning Models](https://mlsanity.com/paper/2510.23191/) (2025)
- [Evaluating AI Evaluation: Perils and Prospects](https://mlsanity.com/paper/2407.09221/) (2024)
- [Measuring the Measurers: Quality Evaluation of Hallucination Benchmarks for Large Vision-Language Models](https://mlsanity.com/paper/2406.17115/) (2024)
- [A validity-guided workflow for robust large language model research in psychology](https://mlsanity.com/paper/2507.04491/) (2025)
- [Survey of Computerized Adaptive Testing: A Machine Learning Perspective](https://mlsanity.com/paper/2404.00712/) (2024)
- [Improving the Validity and Practical Usefulness of AI/ML Evaluations Using an Estimands Framework](https://mlsanity.com/paper/2406.10366/) (2024)
- [Evaluating Implicit Bias in Large Language Models by Attacking From a Psychometric Perspective](https://mlsanity.com/paper/2406.14023/) (2024)
- [Interactive Learning for LLM Reasoning](https://mlsanity.com/paper/2509.26306/) (2025)
- [Cognitive phantoms in LLMs through the lens of latent variables](https://mlsanity.com/paper/2409.15324/) (2024)
- [Measuring What AI Systems Might Do: Towards A Measurement Science in AI](https://mlsanity.com/paper/2603.00063/) (2026)
- [Beyond BFI: The CSI for Enhanced Reliability and Validity in Evaluating LLM Personality Traits](https://mlsanity.com/paper/2503.20182/) (2025)
- [Position: AI Evaluations Should be Grounded on a Theory of Capability](https://mlsanity.com/paper/2509.19590/) (2025)
- [MM-OPERA: Benchmarking Open-ended Association Reasoning for Large Vision-Language Models](https://mlsanity.com/paper/2510.26937/) (2025)
- [Evolutionary Computation for the Design and Enrichment of General-Purpose Artificial Intelligence Systems: Survey and Prospects](https://mlsanity.com/paper/2407.08745/) (2024)
- [FairTree: Subgroup Fairness Auditing of Machine Learning Models with Bias-Variance Decomposition](https://mlsanity.com/paper/2604.19357/) (2026)
- [A Novel Psychometrics-Based Approach to Developing Professional Competency Benchmark for Large Language Models](https://mlsanity.com/paper/2411.00045/) (2024)
- [Principles and Guidelines for Randomized Controlled Trials in AI Evaluation](https://mlsanity.com/paper/2605.02050/) (2026)
- [An Interpretable and Scalable Framework for Evaluating Large Language Models](https://mlsanity.com/paper/2605.07046/) (2026)
- [Harmful Content Is Not Enough: Continuation Framing Moderates In-Context Emergent Misalignment](https://mlsanity.com/paper/2608.08212/) (2026)
- [Accounting for Bias Enables Sustainable LLM Evaluation](https://mlsanity.com/paper/2609.31184/) (2026)

## Abstract (arXiv)

Comprehensive and accurate evaluation of general-purpose AI systems such as large language models allows for effective mitigation of their risks and deepened understanding of their capabilities. Current evaluation methodology, mostly based on benchmarks of specific tasks, falls short of adequately assessing these versatile AI systems, as present techniques lack a scientific foundation for predicting their performance on unforeseen tasks and explaining their varying performance on specific task items or user inputs. Moreover, existing benchmarks of specific tasks raise growing concerns about their reliability and validity. To tackle these challenges, we suggest transitioning from task-oriented evaluation to construct-oriented evaluation. Psychometrics, the science of psychological measurement, provides a rigorous methodology for identifying and measuring the latent constructs that underlie performance across multiple tasks. We discuss its merits, warn against potential pitfalls, and propose a framework to put it into practice. Finally, we explore future opportunities of integrating psychometrics with the evaluation of general-purpose AI systems.