mlsanityData to 14 Oct 2026

Position: Stop Evaluating AI with Human Tests, Develop Principled, AI-specific Tests instead

arXiv 2507.23009 ↗First posted 30 Jul 2025cs.LG cs.AIcited by 7

Not on the map. Why?

Abstract (arXiv)

Large Language Models (LLMs) have achieved remarkable results on a range of standardized tests originally designed to assess human cognitive and psychological traits, such as intelligence and personality. While these results are often interpreted as strong evidence of human-like characteristics in LLMs, this paper argues that such interpretations constitute an ontological error. Human psychological and educational tests are theory-driven measurement instruments, calibrated to a specific human population. Applying these tests to non-human subjects without empirical validation, risks mischaracterizing what is being measured. Furthermore, a growing trend frames AI performance on benchmarks as measurements of traits such as ``intelligence'', despite known issues with validity, data contamination, cultural bias and sensitivity to superficial prompt changes. We argue that interpreting benchmark performance as measurements of human-like traits, lacks sufficient theoretical and empirical justification. This leads to our position: Stop Evaluating AI with Human Tests, Develop Principled, AI-specific Tests instead. We call for the development of principled, AI-specific evaluation frameworks tailored to AI systems. Such frameworks might build on existing frameworks for constructing and validating psychometrics tests, or could be created entirely from scratch to fit the unique context of AI.

Key facts. Cited by 7 papers in the map (data through 14 Oct 2026). Builds on 34 of its 86 references that are in the map. First posted on arXiv 30 Jul 2025.

Builds on

34 of 86 references.

References · 34

  1. 2023GPT-4 Technical Reportcited by 18,396
  2. 2024The Llama 3 Herd of Modelscited by 14,715
  3. 2020Measuring Massive Multitask Language Understandingcited by 7,963
  4. 2020Scaling Laws for Neural Language Modelscited by 6,586
  5. 2018GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understandingcited by 4,899
  6. 2018Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challengecited by 4,886
  7. 2016Equality of Opportunity in Supervised Learningcited by 2,007
  8. 2022Beyond the Imitation Game: Quantifying and extrapolating the capabilities of language modelscited by 1,998
  9. 2024MMLU-Pro: A More Robust and Challenging Multi-Task Language Understanding Benchmarkcited by 1,848
  10. 2022Challenging BIG-Bench Tasks and Whether Chain-of-Thought Can Solve Themcited by 1,807
  11. 2019Do ImageNet Classifiers Generalize to ImageNet?cited by 1,515
  12. 2018Sanity Checks for Saliency Mapscited by 1,156
22 more
  1. 2023Whose Opinions Do Language Models Reflect?cited by 773
  2. 2021Accuracy on the Line: On the Strong Correlation Between Out-of-Distribution and In-Distribution Generalizationcited by 249
  3. 2022Emergent Analogical Reasoning in Large Language Modelscited by 240
  4. 2024Are We Done with MMLU?cited by 194
  5. 2024Global MMLU: Understanding and Addressing Cultural and Linguistic Biases in Multilingual Evaluationcited by 191
  6. 2023Embers of Autoregression: Understanding Large Language Models Through the Problem They are Trained to Solvecited by 178
  7. 2024Benchmark Data Contamination of Large Language Models: A Surveycited by 147
  8. 2024Observational Scaling Laws and the Predictability of Language Model Performancecited by 130
  9. 2025BIG-Bench Extra Hardcited by 125
  10. 2022Evaluating and Inducing Personality in Pre-trained Language Modelscited by 123
  11. 2020The Effect of Natural Distribution Shift on Question Answering Modelscited by 104
  12. 2023Questioning the Survey Responses of Large Language Modelscited by 74
  13. 2025Measurement to Meaning: A Validity-Centered Framework for AI Evaluationcited by 57
  14. 2025Toward an Evaluation Science for Generative AI Systemscited by 45
  15. 2022ID and OOD Performance Are Sometimes Inversely Correlated on Real-world Datasetscited by 44
  16. 2024Changing Answer Order Can Decrease MMLU Accuracycited by 42
  17. 2024Training on the Test Task Confounds Evaluation and Emergencecited by 34
  18. 2023Challenging the Validity of Personality Tests for Large Language Modelscited by 25
  19. 2024ImageNot: A contrast with ImageNet preserves model rankingscited by 14
  20. 2023Effective Robustness against Natural Distribution Shifts for Models with Different Training Datacited by 13
  21. 2024Watching the Generative AI Hype Bubble Deflatecited by 8
  22. 2025Are Domain Generalization Benchmarks with Accuracy on the Line Misspecified?cited by 5

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 2; 2026: 5.

7 papers.

  1. 2025The Benchmarking Epistemology: Validity Theory for Evaluating Machine Learning Modelscited by 13
  2. 2025Do Psychometric Tests Work for Large Language Models? Evaluation of Tests on Sexism, Racism, and Moralitycited by 9
  3. 2026Establishing Construct Validity in LLM Capability Benchmarks Requires Nomological Networkscited by 6
  4. 2026Cutting LLM Evaluation Costs with SySRs: A Bandit Algorithm that Provably Exploits Model Similaritycited by 2
  5. 2026Apparent Psychological Profiles of Large Language Models are Largely a Measurement Artifactcited by 2
  6. 2026Machine individuality: Separating genuine idiosyncrasy from response bias in large language modelscited by 1
  7. 2026AuAu: A Benchmark for Auditing Authoritarian Alignment in Large Language Modelscited by 0