# Establishing Construct Validity in LLM Capability Benchmarks Requires Nomological Networks

arXiv: https://arxiv.org/abs/2603.15121 · HTML: https://mlsanity.com/paper/2603.15121/

## Key facts

- Cited by 6 papers in the map (data through 14 Oct 2026).
- Builds on 16 of its 57 references that are in the map.
- First posted on arXiv 16 Mar 2026.

## Builds on (most cited first, 16 of 16)

- [Locating and Editing Factual Associations in GPT](https://mlsanity.com/paper/2202.05262/) (2022)
- [Do ImageNet Classifiers Generalize to ImageNet?](https://mlsanity.com/paper/1902.10811/) (2019)
- [Open Problems in Mechanistic Interpretability](https://mlsanity.com/paper/2501.16496/) (2025)
- [Can Machines Learn Morality? The Delphi Experiment](https://mlsanity.com/paper/2110.07574/) (2021)
- [Measuring what Matters: Construct Validity in Large Language Model Benchmarks](https://mlsanity.com/paper/2511.04703/) (2025)
- [Language Models Use Trigonometry to Do Addition](https://mlsanity.com/paper/2502.00873/) (2025)
- [Measurement to Meaning: A Validity-Centered Framework for AI Evaluation](https://mlsanity.com/paper/2505.10573/) (2025)
- [Medical Large Language Model Benchmarks Should Prioritize Construct Validity](https://mlsanity.com/paper/2503.10694/) (2025)
- [A Definition of AGI](https://mlsanity.com/paper/2510.18212/) (2025)
- [A Philosophical Introduction to Language Models - Part II: The Way Forward](https://mlsanity.com/paper/2405.03207/) (2024)
- [The Benchmarking Epistemology: Validity Theory for Evaluating Machine Learning Models](https://mlsanity.com/paper/2510.23191/) (2025)
- [Estimating Psychological Networks and their Accuracy: A Tutorial Paper](https://mlsanity.com/paper/1604.08462/) (2016)
- [What is it for a Machine Learning Model to Have a Capability?](https://mlsanity.com/paper/2405.08989/) (2024)
- [Position: Stop Evaluating AI with Human Tests, Develop Principled, AI-specific Tests instead](https://mlsanity.com/paper/2507.23009/) (2025)
- [The Costs of Pretending That There Are Data-Generating Probability Distributions in the Social World](https://mlsanity.com/paper/2407.17395/) (2024)
- [Artificial Neural Nets and the Representation of Human Concepts](https://mlsanity.com/paper/2312.05337/) (2023)

## Built on by (most cited first, 6 of 6)

- [Correct codes for the wrong reasons? validating LLMs as measurement instruments for theoretical constructs](https://mlsanity.com/paper/2606.28574/) (2026)
- [The yes-no bias of large language models reflects answer order and wording, not shifts in moral judgment](https://mlsanity.com/paper/2607.05552/) (2026)
- [Exposure is not manifestation: measurement target and output resolution jointly determine which behavioural-faithfulness evaluator wins](https://mlsanity.com/paper/2607.09306/) (2026)
- [When benchmark inferences do not compose: Projectibility in AI evaluation](https://mlsanity.com/paper/2607.26159/) (2026)
- [The Guard That Cried Wolf: How Scary Words Make Agent Guardrails Refuse Legitimate Actions](https://mlsanity.com/paper/2608.27009/) (2026)
- [Does Target Alignment Mean Target Recovery? An Evidence-Ladder Study of Adversarial Claims on Contrastive Encoders](https://mlsanity.com/paper/2610.11938/) (2026)

## Abstract (arXiv)

Recent work in machine learning increasingly attributes human-like capabilities such as reasoning or theory of mind to large language models (LLMs) on the basis of benchmark performance. This paper examines this practice through the lens of construct validity, understood as the problem of linking theoretical capabilities to their empirical measurements. It contrasts three influential frameworks: the nomological account developed by Cronbach and Meehl, the inferential account proposed by Messick and refined by Kane, and Borsboom's causal account. I argue that the nomological account provides the most suitable foundation for current LLM capability research. It avoids the strong ontological commitments of the causal account while offering a more substantive framework for articulating construct meaning than the inferential account. I explore the conceptual implications of adopting the nomological account for LLM research through a concrete case: the assessment of reasoning capabilities in LLMs.