# Position: AI Evaluation Should Learn from How We Test Humans

arXiv: https://arxiv.org/abs/2306.10512 · HTML: https://mlsanity.com/paper/2306.10512/

## Key facts

- Cited by 36 papers in the map (data through 14 Oct 2026).
- Builds on 29 of its 126 references that are in the map.
- First posted on arXiv 18 Jun 2023.

## Builds on (most cited first, 29 of 29)

- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [Measuring Massive Multitask Language Understanding](https://mlsanity.com/paper/2009.03300/) (2020)
- [Measuring Mathematical Problem Solving With the MATH Dataset](https://mlsanity.com/paper/2103.03874/) (2021)
- [PaLM: Scaling Language Modeling with Pathways](https://mlsanity.com/paper/2204.02311/) (2022)
- [Model-Agnostic Meta-Learning for Fast Adaptation of Deep Networks](https://mlsanity.com/paper/1703.03400/) (2017)
- [SQuAD: 100,000+ Questions for Machine Comprehension of Text](https://mlsanity.com/paper/1606.05250/) (2016)
- [Finetuned Language Models Are Zero-Shot Learners](https://mlsanity.com/paper/2109.01652/) (2021)
- [Can a Suit of Armor Conduct Electricity? A New Dataset for Open Book Question Answering](https://mlsanity.com/paper/1809.02789/) (2018)
- [Know What You Don't Know: Unanswerable Questions for SQuAD](https://mlsanity.com/paper/1806.03822/) (2018)
- [What Disease does this Patient Have? A Large-scale Open Domain Question Answering Dataset from Medical Exams](https://mlsanity.com/paper/2009.13081/) (2020)
- [A Multitask, Multilingual, Multimodal Evaluation of ChatGPT on Reasoning, Hallucination, and Interactivity](https://mlsanity.com/paper/2302.04023/) (2023)
- [Adversarial NLI: A New Benchmark for Natural Language Understanding](https://mlsanity.com/paper/1910.14599/) (2019)
- [The NarrativeQA Reading Comprehension Challenge](https://mlsanity.com/paper/1712.07040/) (2017)
- [Toxicity in ChatGPT: Analyzing Persona-assigned Language Models](https://mlsanity.com/paper/2304.05335/) (2023)
- [Speak, Memory: An Archaeology of Books Known to ChatGPT/GPT-4](https://mlsanity.com/paper/2305.00118/) (2023)
- [EKT: Exercise-aware Knowledge Tracing for Student Performance Prediction](https://mlsanity.com/paper/1906.05658/) (2019)
- [Revisiting Out-of-distribution Robustness in NLP: Benchmark, Analysis, and LLMs Evaluations](https://mlsanity.com/paper/2306.04618/) (2023)
- [On the Planning Abilities of Large Language Models (A Critical Investigation with a Proposed Benchmark)](https://mlsanity.com/paper/2302.06706/) (2023)
- [Have LLMs Advanced Enough? A Challenging Problem Solving Benchmark For Large Language Models](https://mlsanity.com/paper/2305.15074/) (2023)
- [Comparing Test Sets with Item Response Theory](https://mlsanity.com/paper/2106.00840/) (2021)
- [Dataset Distillation with Convexified Implicit Gradients](https://mlsanity.com/paper/2302.06755/) (2023)
- [RAFT: A Real-World Few-Shot Text Classification Benchmark](https://mlsanity.com/paper/2109.14076/) (2021)
- [Large Language Models as Tax Attorneys: A Case Study in Legal Capabilities Emergence](https://mlsanity.com/paper/2306.07075/) (2023)
- [On Robustness of Prompt-based Semantic Parsing with Large Pre-trained Language Model: An Empirical Study on Codex](https://mlsanity.com/paper/2301.12868/) (2023)
- [How Predictable Are Large Language Model Capabilities? A Case Study on BIG-bench](https://mlsanity.com/paper/2305.14947/) (2023)
- [Variational Item Response Theory: Fast, Accurate, and Expressive](https://mlsanity.com/paper/2002.00276/) (2020)
- [Leveraging Large Language Models for Topic Classification in the Domain of Public Affairs](https://mlsanity.com/paper/2306.02864/) (2023)
- [Stabilizing Q-learning with Linear Architectures for Provably Efficient Learning](https://mlsanity.com/paper/2206.00796/) (2022)
- [A Two-Stage Active Learning Algorithm for k-Nearest Neighbors](https://mlsanity.com/paper/2211.10773/) (2022)

## Built on by (most cited first, 30 of 36)

- [A Survey on Evaluation of Large Language Models](https://mlsanity.com/paper/2307.03109/) (2023)
- [tinyBenchmarks: evaluating LLMs with fewer examples](https://mlsanity.com/paper/2402.14992/) (2024)
- [Evaluating Mathematical Reasoning of Large Language Models: A Focus on Error Identification and Correction](https://mlsanity.com/paper/2406.00755/) (2024)
- [metabench -- A Sparse Benchmark of Reasoning and Knowledge in Large Language Models](https://mlsanity.com/paper/2407.12844/) (2024)
- [When does In-context Learning Fall Short and Why? A Study on Specification-Heavy Tasks](https://mlsanity.com/paper/2311.08993/) (2023)
- [Surveying the Effects of Quality, Diversity, and Complexity in Synthetic Data From Large Language Models](https://mlsanity.com/paper/2412.02980/) (2024)
- [Who is ChatGPT? Benchmarking LLMs' Psychological Portrayal Using PsychoBench](https://mlsanity.com/paper/2310.01386/) (2023)
- [Adaptive Testing for LLM Evaluation: A Psychometric Alternative to Static Benchmarks](https://mlsanity.com/paper/2511.04689/) (2025)
- [FFT: Towards Harmlessness Evaluation and Analysis for LLMs with Factuality, Fairness, Toxicity](https://mlsanity.com/paper/2311.18580/) (2023)
- [Beyond Static Datasets: A Deep Interaction Approach to LLM Evaluation](https://mlsanity.com/paper/2309.04369/) (2023)
- [The Confidence-Competence Gap in Large Language Models: A Cognitive Study](https://mlsanity.com/paper/2309.16145/) (2023)
- [A Survey of Models for Cognitive Diagnosis: New Developments and Future Directions](https://mlsanity.com/paper/2407.05458/) (2024)
- [A Cognitive Evaluation Benchmark of Image Reasoning and Description for Large Vision-Language Models](https://mlsanity.com/paper/2402.18409/) (2024)
- [Survey of Computerized Adaptive Testing: A Machine Learning Perspective](https://mlsanity.com/paper/2404.00712/) (2024)
- [Toward a unified framework for data-efficient evaluation of large language models](https://mlsanity.com/paper/2510.04051/) (2025)
- [BenchMarker: An Education-Inspired Toolkit for Highlighting Flaws in Multiple-Choice Benchmarks](https://mlsanity.com/paper/2602.06221/) (2026)
- [The Artificial Intelligence Cognitive Examination: A Survey on the Evolution of Multimodal Evaluation from Recognition to Reasoning](https://mlsanity.com/paper/2510.04141/) (2025)
- [Automated Benchmark Generation from Domain Guidelines Informed by Bloom's Taxonomy](https://mlsanity.com/paper/2601.20253/) (2026)
- [GPT-4's assessment of its performance in a USMLE-based case study](https://mlsanity.com/paper/2402.09654/) (2024)
- [Development of Cognitive Intelligence in Pre-trained Language Models](https://mlsanity.com/paper/2407.01047/) (2024)
- [MM-OPERA: Benchmarking Open-ended Association Reasoning for Large Vision-Language Models](https://mlsanity.com/paper/2510.26937/) (2025)
- [Rich Insights from Cheap Signals: Efficient Evaluations via Tensor Factorization](https://mlsanity.com/paper/2603.02029/) (2026)
- [Select Smarter, Not More: Prompt-Aware Evaluation Scheduling with Submodular Guarantees](https://mlsanity.com/paper/2604.11328/) (2026)
- [FairTree: Subgroup Fairness Auditing of Machine Learning Models with Bias-Variance Decomposition](https://mlsanity.com/paper/2604.19357/) (2026)
- [Solipsistic Superintelligence is Unlikely to be Cooperative](https://mlsanity.com/paper/2606.03237/) (2026)
- [Every Wrong Answer Counts: Option-Level Psychometrics for LLM Multiple-Choice Benchmarks](https://mlsanity.com/paper/2608.02966/) (2026)
- [LLM Evaluation on Unseen Questions: Contextual Multidimensional IRT Model](https://mlsanity.com/paper/2608.22295/) (2026)
- [Real-World AI Evaluation: How FRAME Generates Systematic Evidence to Resolve the Decision-Maker's Dilemma](https://mlsanity.com/paper/2603.13294/) (2026)
- [Leveraging Computerized Adaptive Testing for Cost-effective Evaluation of Large Language Models in Medical Benchmarking](https://mlsanity.com/paper/2603.23506/) (2026)
- [CoFEE: Reasoning Control for LLM-Based Feature Discovery](https://mlsanity.com/paper/2604.21584/) (2026)

## Abstract (arXiv)

As AI systems continue to evolve, their rigorous evaluation becomes crucial for their development and deployment. Researchers have constructed various large-scale benchmarks to determine their capabilities, typically against a gold-standard test set and report metrics averaged across all items. However, this static evaluation paradigm increasingly shows its limitations, including high evaluation costs, data contamination, and the impact of low-quality or erroneous items on evaluation reliability and efficiency. In this Position, drawing from human psychometrics, we discuss a paradigm shift from static evaluation methods to adaptive testing. This involves estimating the characteristics or value of each test item in the benchmark, and tailoring each model's evaluation instead of relying on a fixed test set. This paradigm provides robust ability estimation, uncovering the latent traits underlying a model's observed scores. This position paper analyze the current possibilities, prospects, and reasons for adopting psychometrics in AI evaluation. We argue that psychometrics, a theory originating in the 20th century for human assessment, could be a powerful solution to the challenges in today's AI evaluations.