# Medical Large Language Model Benchmarks Should Prioritize Construct Validity

arXiv: https://arxiv.org/abs/2503.10694 · HTML: https://mlsanity.com/paper/2503.10694/

## Key facts

- Cited by 38 papers in the map (data through 14 Oct 2026).
- Builds on 11 of its 58 references that are in the map.
- First posted on arXiv 12 Mar 2025.

## Builds on (most cited first, 11 of 11)

- [Training Verifiers to Solve Math Word Problems](https://mlsanity.com/paper/2110.14168/) (2021)
- [ImageNet Large Scale Visual Recognition Challenge](https://mlsanity.com/paper/1409.0575/) (2014)
- [Emergent Abilities of Large Language Models](https://mlsanity.com/paper/2206.07682/) (2022)
- [Large Language Models Encode Clinical Knowledge](https://mlsanity.com/paper/2212.13138/) (2022)
- [What Disease does this Patient Have? A Large-scale Open Domain Question Answering Dataset from Medical Exams](https://mlsanity.com/paper/2009.13081/) (2020)
- [Large Language Models Can Be Easily Distracted by Irrelevant Context](https://mlsanity.com/paper/2302.00093/) (2023)
- [AI and the Everything in the Whole Wide World Benchmark](https://mlsanity.com/paper/2111.15366/) (2021)
- [Adapted Large Language Models Can Outperform Medical Experts in Clinical Text Summarization](https://mlsanity.com/paper/2309.07430/) (2023)
- [Towards Accurate Differential Diagnosis with Large Language Models](https://mlsanity.com/paper/2312.00164/) (2023)
- [Diagnostic Reasoning Prompts Reveal the Potential for Large Language Model Interpretability in Medicine](https://mlsanity.com/paper/2308.06834/) (2023)
- [Does progress on ImageNet transfer to real-world datasets?](https://mlsanity.com/paper/2301.04644/) (2023)

## Built on by (most cited first, 30 of 38)

- [Measuring what Matters: Construct Validity in Large Language Model Benchmarks](https://mlsanity.com/paper/2511.04703/) (2025)
- [ER-Reason: A Benchmark Dataset for LLM Clinical Reasoning in the Emergency Room](https://mlsanity.com/paper/2505.22919/) (2025)
- [Assessing the Chemical Intelligence of Large Language Models](https://mlsanity.com/paper/2505.07735/) (2025)
- [Resolution Diagnostics for Paired LLM Evaluation](https://mlsanity.com/paper/2605.30315/) (2026)
- [Establishing Construct Validity in LLM Capability Benchmarks Requires Nomological Networks](https://mlsanity.com/paper/2603.15121/) (2026)
- [On Path to Multimodal Historical Reasoning: HistBench and HistAgent](https://mlsanity.com/paper/2505.20246/) (2025)
- [Expert Evaluation of Clinical AI Tools on Real Point-of-Care Clinical Queries](https://mlsanity.com/paper/2606.28960/) (2026)
- [VERA-MH Concept Paper](https://mlsanity.com/paper/2510.15297/) (2025)
- [ReX-MLE: The Autonomous Agent Benchmark for Medical Imaging Challenges](https://mlsanity.com/paper/2512.17838/) (2025)
- [The Persona Paradox: Medical Personas as Behavioral Priors in Clinical Language Models](https://mlsanity.com/paper/2601.05376/) (2026)
- [When the Domain Expert Has No Time and the LLM Developer Has No Clinical Expertise: Real-World Lessons from LLM Co-Design in a Safety-Net Hospital](https://mlsanity.com/paper/2508.08504/) (2025)
- [Reliable agent engineering should integrate machine-compatible organizational principles](https://mlsanity.com/paper/2512.07665/) (2025)
- [ContractBench: Can LLM Agents Preserve Observation Contracts?](https://mlsanity.com/paper/2605.17281/) (2026)
- [A Typology of Synthetic Datasets for Dialogue Processing in Clinical Contexts](https://mlsanity.com/paper/2505.03025/) (2025)
- [Measurement as Bricolage: Examining How Data Scientists Construct Target Variables for Predictive Modeling Tasks](https://mlsanity.com/paper/2507.02819/) (2025)
- [Measuring Competency, Not Performance: Item-Aware Evaluation Across Medical Benchmarks](https://mlsanity.com/paper/2509.24186/) (2025)
- [BenchBrowser: Retrieving Evidence for Evaluating Benchmark Validity](https://mlsanity.com/paper/2603.18019/) (2026)
- [Distributional Open-Ended Evaluation of LLM Cultural Value Alignment Based on Value Codebook](https://mlsanity.com/paper/2604.06210/) (2026)
- [AI-Assisted Systematization for Evaluating GenAI Systems](https://mlsanity.com/paper/2605.26001/) (2026)
- [Aligning Clinical Needs and AI Capabilities: A Survey on LLMs for Medical Reasoning](https://mlsanity.com/paper/2607.07761/) (2026)
- [Do Assessment Instruments Measure the Same Thing for Humans and LLMs? A Latent Structure Analysis](https://mlsanity.com/paper/2608.15630/) (2026)
- [MTDiag: A Multi-Turn Diagnostic Dataset Towards Clinically Meaningful LLM Evaluation](https://mlsanity.com/paper/2608.25085/) (2026)
- [DeepQuestion: Systematic Generation of Real-World Challenges for Evaluating LLMs Performance](https://mlsanity.com/paper/2505.24532/) (2025)
- [Aligning Language Model Benchmarks with Pairwise Preferences](https://mlsanity.com/paper/2602.02898/) (2026)
- [Who Benchmarks the Benchmarks? A Case Study of LLM Evaluation in Icelandic](https://mlsanity.com/paper/2603.16406/) (2026)
- [DepthCharge: A Domain-Agnostic Framework for Measuring Depth-Dependent Knowledge in Large Language Models](https://mlsanity.com/paper/2603.23514/) (2026)
- [VERA-MH: Validation of Ethical and Responsible AI in Mental Health](https://mlsanity.com/paper/2605.13318/) (2026)
- [Healthcare LLM Benchmarks Are Only as Good as Their Explicit Assumptions](https://mlsanity.com/paper/2605.22612/) (2026)
- [Evaluating the Diagnostic Robustness of Vision-Language Models Under Visual and Textual Perturbations](https://mlsanity.com/paper/2608.04885/) (2026)
- [Position: Reasoning is a Learnable Rule-Based Process](https://mlsanity.com/paper/2608.12325/) (2026)

## Abstract (arXiv)

Medical large language models (LLMs) research often makes bold claims, from encoding clinical knowledge to reasoning like a physician. These claims are usually backed by evaluation on competitive benchmarks; a tradition inherited from mainstream machine learning. But how do we separate real progress from a leaderboard flex? Medical LLM benchmarks, much like those in other fields, are arbitrarily constructed using medical licensing exam questions. For these benchmarks to truly measure progress, they must accurately capture the real-world tasks they aim to represent. In this position paper, we argue that medical LLM benchmarks should (and indeed can) be empirically evaluated for their construct validity. In the psychological testing literature, "construct validity" refers to the ability of a test to measure an underlying "construct", that is the actual conceptual target of evaluation. By drawing an analogy between LLM benchmarks and psychological tests, we explain how frameworks from this field can provide empirical foundations for validating benchmarks. To put these ideas into practice, we use real-world clinical data in proof-of-concept experiments to evaluate popular medical LLM benchmarks and report significant gaps in their construct validity. Finally, we outline a vision for a new ecosystem of medical LLM evaluation centered around the creation of valid benchmarks.