# Measuring What AI Systems Might Do: Towards A Measurement Science in AI

arXiv: https://arxiv.org/abs/2603.00063 · HTML: https://mlsanity.com/paper/2603.00063/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Builds on 21 of its 60 references that are in the map.
- First posted on arXiv 10 Feb 2026.

## Builds on (most cited first, 21 of 21)

- [GPT-4 Technical Report](https://mlsanity.com/paper/2303.08774/) (2023)
- [Evaluating Large Language Models Trained on Code](https://mlsanity.com/paper/2107.03374/) (2021)
- [Measuring Mathematical Problem Solving With the MATH Dataset](https://mlsanity.com/paper/2103.03874/) (2021)
- [Mechanistic Interpretability for AI Safety -- A Review](https://mlsanity.com/paper/2404.14082/) (2024)
- [Using cognitive psychology to understand GPT-3](https://mlsanity.com/paper/2206.14576/) (2022)
- [tinyBenchmarks: evaluating LLMs with fewer examples](https://mlsanity.com/paper/2402.14992/) (2024)
- [Open Problems in Mechanistic Interpretability](https://mlsanity.com/paper/2501.16496/) (2025)
- [Agentic Misalignment: How LLMs Could Be Insider Threats](https://mlsanity.com/paper/2510.05179/) (2025)
- [Measuring what Matters: Construct Validity in Large Language Model Benchmarks](https://mlsanity.com/paper/2511.04703/) (2025)
- [A Practical Review of Mechanistic Interpretability for Transformer-Based Language Models](https://mlsanity.com/paper/2407.02646/) (2024)
- [Red-Teaming for Generative AI: Silver Bullet or Security Theater?](https://mlsanity.com/paper/2401.15897/) (2024)
- [metabench -- A Sparse Benchmark of Reasoning and Knowledge in Large Language Models](https://mlsanity.com/paper/2407.12844/) (2024)
- [General Scales Unlock AI Evaluation with Explanatory and Predictive Power](https://mlsanity.com/paper/2503.06378/) (2025)
- [Evaluating General-Purpose AI with Psychometrics](https://mlsanity.com/paper/2310.16379/) (2023)
- [How to evaluate control measures for LLM agents? A trajectory from today to superintelligence](https://mlsanity.com/paper/2504.05259/) (2025)
- [Lessons from a Chimp: AI "Scheming" and the Quest for Ape Language](https://mlsanity.com/paper/2507.03409/) (2025)
- [Safety by Measurement: A Systematic Literature Review of AI Safety Evaluation Methods](https://mlsanity.com/paper/2505.05541/) (2025)
- [Inferring Capabilities from Task Performance with Bayesian Triangulation](https://mlsanity.com/paper/2309.11975/) (2023)
- [What is it for a Machine Learning Model to Have a Capability?](https://mlsanity.com/paper/2405.08989/) (2024)
- [Network Psychometrics](https://mlsanity.com/paper/1609.02818/) (2016)
- [Position: AI Evaluations Should be Grounded on a Theory of Capability](https://mlsanity.com/paper/2509.19590/) (2025)

## Built on by (most cited first, 3 of 3)

- [LLMs Can Leak Training Data But Do They Want To? A Propensity-Aware Evaluation of Memorization in LLMs](https://mlsanity.com/paper/2606.06286/) (2026)
- [Item Response Theory for AI Safety](https://mlsanity.com/paper/2608.05086/) (2026)
- [Signals in the Noise: Open Source Intelligence (OSINT) for AI Loss of Control Detection](https://mlsanity.com/paper/2606.20610/) (2026)

## Abstract (arXiv)

Scientists, policy-makers, business leaders, and members of the public care about what modern artificial intelligence systems are disposed to do. Yet terms such as capabilities, propensities, skills, values, and abilities are routinely used interchangeably and conflated with observable performance, with AI evaluation practices rarely specifying what quantity they purport to measure. We argue that capabilities and propensities are dispositional properties - stable features of systems characterised by counterfactual relationships between contextual conditions and behavioural outputs. Measuring a disposition requires (i) hypothesising which contextual properties are causally relevant, (ii) independently operationalising and measuring those properties, and (iii) empirically mapping how variation in those properties affects the probability of the behaviour. Dominant approaches to AI evaluation, from benchmark averages to data-driven latent-variable models such as Item Response Theory, bypass these steps entirely. Building on ideas from philosophy of science, measurement theory, and cognitive science, we develop a principled account of AI capabilities and propensities as dispositions, show why prevailing evaluation practices fail to measure them, and outline what disposition-respecting, scientifically defensible AI evaluation would require.