mlsanityData to 14 Oct 2026

Measuring what Matters: Construct Validity in Large Language Model Benchmarks

arXiv 2511.04703 ↗First posted 3 Nov 2025cs.CL cs.AIcited by 119

Not on the map. Why?

Abstract (arXiv)

Evaluating large language models (LLMs) is crucial for both assessing their capabilities and identifying safety or robustness issues prior to deployment. Reliably measuring abstract and complex phenomena such as 'safety' and 'robustness' requires strong construct validity, that is, having measures that represent what matters to the phenomenon. With a team of 29 expert reviewers, we conduct a systematic review of 445 LLM benchmarks from leading conferences in natural language processing and machine learning. Across the reviewed articles, we find patterns related to the measured phenomena, tasks, and scoring metrics which undermine the validity of the resulting claims. To address these shortcomings, we provide eight key recommendations and detailed actionable guidance to researchers and practitioners in developing LLM benchmarks.

Key facts. Cited by 119 papers in the map (data through 14 Oct 2026). Builds on 56 of its 73 references that are in the map. First posted on arXiv 3 Nov 2025.

Builds on

56 of 73 references.

References · 56

  1. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,459
  2. 2021Training Verifiers to Solve Math Word Problemscited by 10,109
  3. 2023Sparks of Artificial General Intelligence: Early experiments with GPT-4cited by 2,592
  4. 2022Emergent Abilities of Large Language Modelscited by 2,512
  5. 2019SuperGLUE: A Stickier Benchmark for General-Purpose Language Understanding Systemscited by 1,729
  6. 2018Datasheets for Datasetscited by 1,386
  7. 2023AgentBench: Evaluating LLMs as Agentscited by 1,267
  8. 2022WebShop: Towards Scalable Real-World Web Interaction with Grounded Language Agentscited by 1,251
  9. 2019Right for the Wrong Reasons: Diagnosing Syntactic Heuristics in Natural Language Inferencecited by 876
  10. 2023Plan-and-Solve Prompting: Improving Zero-Shot Chain-of-Thought Reasoning by Large Language Modelscited by 560
  11. 2024GSM-Symbolic: Understanding the Limitations of Mathematical Reasoning in Large Language Modelscited by 550
  12. 2023Are Emergent Abilities of Large Language Models a Mirage?cited by 523
38 more
  1. 2021Dynabench: Rethinking Benchmarking in NLPcited by 459
  2. 2021AI and the Everything in the Whole Wide World Benchmarkcited by 349
  3. 2023State of What Art? A Call for Multi-Prompt LLM Evaluationcited by 310
  4. 2024tinyBenchmarks: evaluating LLMs with fewer examplescited by 310
  5. 2024LiveBench: A Challenging, Contamination-Limited LLM Benchmarkcited by 262
  6. 2023SciBench: Evaluating College-Level Scientific Problem-Solving Abilities of Large Language Modelscited by 256
  7. 2021What Will it Take to Fix Benchmarking in Natural Language Understanding?cited by 203
  8. 2023Don't Make Your LLM an Evaluation Benchmark Cheatercited by 194
  9. 2024A Careful Examination of Large Language Model Performance on Grade School Arithmeticcited by 191
  10. 2024Global MMLU: Understanding and Addressing Cultural and Linguistic Biases in Multilingual Evaluationcited by 191
  11. 2024Lessons from the Trenches on Reproducible Evaluation of Language Modelscited by 179
  12. 2023Embers of Autoregression: Understanding Large Language Models Through the Problem They are Trained to Solvecited by 178
  13. 2023FANToM: A Benchmark for Stress-testing Machine Theory of Mind in Interactionscited by 171
  14. 2024ProSA: Assessing and Understanding the Prompt Sensitivity of LLMscited by 149
  15. 2024Adding Error Bars to Evals: A Statistical Approach to Language Model Evaluationscited by 148
  16. 2024The PRISM Alignment Dataset: What Participatory, Representative and Individualised Human Feedback Reveals About the Subjective and Multicultural Alignment of Large Language Modelscited by 144
  17. 2024Observational Scaling Laws and the Predictability of Language Model Performancecited by 130
  18. 2024Evaluating Frontier Models for Dangerous Capabilitiescited by 128
  19. 2024Tokenization counts: the impact of tokenization on arithmetic in frontier LLMscited by 115
  20. 2024BetterBench: Assessing AI Benchmarks, Uncovering Issues, and Establishing Best Practicescited by 114
  21. 2024LLM Dataset Inference: Did you train on my dataset?cited by 110
  22. 2024Inadequacies of Large Language Model Benchmarks in the Era of Generative Artificial Intelligencecited by 68
  23. 2024DA-Code: Agent Data Science Code Generation Benchmark for Large Language Modelscited by 58
  24. 2023Dr.Spider: A Diagnostic Evaluation Benchmark towards Text-to-SQL Robustnesscited by 53
  25. 2024LitSearch: A Retrieval Benchmark for Scientific Literature Searchcited by 50
  26. 2022Evaluation Gaps in Machine Learning Practicecited by 49
  27. 2023Embrace Divergence for Richer Insights: A Multi-document Summarization Benchmark and a Case Study on Summarizing Diverse Information from News Articlescited by 45
  28. 2025Toward an Evaluation Science for Generative AI Systemscited by 45
  29. 2024WikiContradict: A Benchmark for Evaluating LLMs on Real-World Knowledge Conflicts from Wikipediacited by 44
  30. 2023Revealing the structure of language model capabilitiescited by 40
  31. 2025Medical Large Language Model Benchmarks Should Prioritize Construct Validitycited by 38
  32. 2024IndiBias: A Benchmark Dataset to Measure Social Biases in Language Models for Indian Contextcited by 32
  33. 2025General Scales Unlock AI Evaluation with Explanatory and Predictive Powercited by 31
  34. 2024LINGOLY: A Benchmark of Olympiad-Level Linguistic Reasoning Puzzles in Low-Resource and Extinct Languagescited by 28
  35. 2023Evidence of interrelated cognitive-like capabilities in large language models: Indications of artificial general intelligence or achievement?cited by 27
  36. 2019Value-laden Disciplinary Shifts in Machine Learningcited by 25
  37. 2021"Garbage In, Garbage Out" Revisited: What Do Machine Learning Application Papers Report About Human-Labeled Training Data?cited by 24
  38. 2022WikiWhy: Answering and Explaining Cause-and-Effect Questionscited by 21

50 most cited of 56 All 56 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 1; 2026: 118.

119 papers.

Sort:

By area:

  1. 2026Beyond Benchmarks: MathArena as an Evaluation Platform for Mathematics with LLMscited by 137
  2. 2026Brittlebench: Quantifying LLM robustness via prompt sensitivitycited by 21
  3. 2026When Tools Fail: Benchmarking Dynamic Replanning and Anomaly Recovery in LLM Agentscited by 17
  4. 2025Eval Factsheets: A Structured Framework for Documenting AI Evaluationscited by 9
  5. 2026One Persona, Many Cues, Different Results: How Sociodemographic Cues Impact LLM Personalizationcited by 9
  6. 2026Autonomous Research Agents: A Survey of AI Scientists and the Verification Gapcited by 9
  7. 2026Training AI Scientists to Replicate Researchcited by 9
  8. 2026Resolution Diagnostics for Paired LLM Evaluationcited by 7
  9. 2026Position: Anthropomorphic Misalignment Research Needs Stronger Evidencecited by 7
  10. 2026Establishing Construct Validity in LLM Capability Benchmarks Requires Nomological Networkscited by 6
  11. 2026ProcCtrlBench: Evaluating Process-Level Defects and Control Preservation in LLM Coding Agentscited by 6
  12. 2026Beyond the Mean: Within-Model Reliable Change Detection for LLM Evaluationcited by 5
38 more
  1. 2026Designing Benchmarks for Knowledge Workcited by 5
  2. 2026JudgmentBench: Comparing Rubric and Preference Evaluation for Quality Assessmentcited by 5
  3. 2026The Implementation Lottery: Auditing Idea Reliability in Automated Researchcited by 5
  4. 2026Different Demographic Cues Yield Inconsistent Conclusions About LLM Personalization and Biascited by 4
  5. 2026Measuring Faithfulness Depends on How You Measure: Classifier Sensitivity in LLM Chain-of-Thought Evaluationcited by 4
  6. 2026Evaluation Cards: An Interpretive Layer for AI Evaluation Reportingcited by 4
  7. 2026Agent Benchmarks Fail Public Sector Requirementscited by 3
  8. 2026Language Statistics and False Belief Reasoning: Evidence from 41 Open-Weight LMscited by 3
  9. 2026Measuring What AI Systems Might Do: Towards A Measurement Science in AIcited by 3
  10. 2026AI Evaluation Should Require Standardized Item-Level Data Releasescited by 3
  11. 2026Engineering Reliable Coding Agents: Evaluating and Operating the System Around the Modelcited by 3
  12. 2026pdfQA: Diverse, Challenging, and Realistic Question Answering over PDFscited by 2
  13. 2026CIRCLE: A Framework for Evaluating AI from a Real-World Lenscited by 2
  14. 2026pAI/MSc: ML Theory Research with Humans on the Loopcited by 2
  15. 2026NLG Evaluation: Past, Present, Futurecited by 2
  16. 2026When Behavioral Safety Evaluation Fails: A Representation-Level Perspectivecited by 2
  17. 2026Correct codes for the wrong reasons? validating LLMs as measurement instruments for theoretical constructscited by 2
  18. 2026Safety, or Just Capability? A Validity Audit of Agent-Safety Benchmarkscited by 2
  19. 2026Don't Claim Benchmark-Oriented Optimization Improves General Coding Capability -- Diverse Evaluation Is Requiredcited by 2
  20. 2026K-Bench: measuring model performance on real scientific agent requestscited by 2
  21. 2026When Guardrails Look Effective: Construct Validity Failures in LLM Agent Commerce Evaluationcited by 2
  22. 2026Do Agent Benchmarks Do What They Say? An Executable-Contract Audit of Tool-Using Agent Environmentscited by 2
  23. 2026The Algorithmic Gaze of Image Quality Assessment: An Audit and Trace Ethnography of the LAION-Aesthetics Predictorcited by 1
  24. 2026Benchmarking at the Edge of Comprehensioncited by 1
  25. 2026Making AI Evaluation Deployment Relevant Through Context Specificationcited by 1
  26. 2026BenchBrowser: Retrieving Evidence for Evaluating Benchmark Validitycited by 1
  27. 2026ClimateCause: Complex and Implicit Causal Structures in Climate Reportscited by 1
  28. 2026MADE: A Living Benchmark for Multi-Label Text Classification with Uncertainty Quantification of Medical Device Adverse Eventscited by 1
  29. 2026When No Benchmark Exists: Validating Comparative LLM Safety Scoring Without Ground-Truth Labelscited by 1
  30. 2026Do LLMs Hold Their Values? MANTA: A Multi-Turn Adversarial Benchmark for Animal Welfare Reasoningcited by 1
  31. 2026AI-Assisted Systematization for Evaluating GenAI Systemscited by 1
  32. 2026Beyond Questions: Evaluating the Stated Knowledge of LLMscited by 1
  33. 2026Human Capital, Not Model Benchmarks, Predicts Hybrid Intelligence in Forecastingcited by 1
  34. 2026The yes-no bias of large language models reflects answer order and wording, not shifts in moral judgmentcited by 1
  35. 2026Instrument Effects in Language-Model Honesty Evaluation: An Auditable Single-System Demonstrationcited by 1
  36. 2026MORFES: A Benchmark for Productive Inflectional Competence in Modern Greekcited by 1
  37. 2026Rethinking Automated Program Repair: The Impact of Bug Complexity, Fault Localization, and LLM Cost-efficiencycited by 1
  38. 2026ContractScrub: A benchmark for final review of legal contractscited by 1

50 most cited of 119 All 119 (JSON)