mlsanityData to 7 Oct 2026

The Benchmarking Epistemology: Validity Theory for Evaluating Machine Learning Models

arXiv 2510.23191 ↗First posted 27 Oct 2025cs.LG stat.MLcited by 12

Not on the map. Why?

Abstract (arXiv)

Predictive benchmarking, evaluating machine learning models based on predictive performance and competitive ranking, is central to machine learning research and scientific inquiry. However, benchmark scores at best measure performance relative to a specific dataset and learning problem. Drawing substantial scientific inferences requires additional assumptions. Adapting ideas from psychological validity theory, we propose validity conditions that make these assumptions explicit. In two case studies---ImageNet and the Fragile Families Challenge---we show how benchmark results can support inferences about research progress and limits of predictability, situating predictive benchmarking as a distinct epistemic practice in machine learning.

Key facts. Cited by 12 papers in the map (data through 7 Oct 2026). Builds on 47 of its 137 references that are in the map. First posted on arXiv 27 Oct 2025.

Builds on

47 of 137 references.

References · 47

  1. 2014ImageNet Large Scale Visual Recognition Challengecited by 8,441
  2. 2014Explaining and Harnessing Adversarial Examplescited by 7,287
  3. 2015Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shiftcited by 6,979
  4. 2019Benchmarking Neural Network Robustness to Common Corruptions and Perturbationscited by 2,857
  5. 2019SuperGLUE: A Stickier Benchmark for General-Purpose Language Understanding Systemscited by 1,718
  6. 2019Do ImageNet Classifiers Generalize to ImageNet?cited by 1,507
  7. 2022ChartQA: A Benchmark for Question Answering about Charts with Visual and Logical Reasoningcited by 1,449
  8. 2019CheXpert: A Large Chest Radiograph Dataset with Uncertainty Labels and Expert Comparisoncited by 1,410
  9. 2018Do Better ImageNet Models Transfer Better?cited by 560
  10. 2021AI and the Everything in the Whole Wide World Benchmarkcited by 340
  11. 2020Are we done with ImageNet?cited by 329
  12. 2020WeatherBench: A benchmark dataset for data-driven weather forecastingcited by 272
35 more
  1. 2020Data and its (dis)contents: A survey of dataset development and use in machine learning researchcited by 259
  2. 2018Generalisation in humans and deep neural networkscited by 237
  3. 2022Talking About Large Language Modelscited by 174
  4. 2023Clever Hans or Neural Theory of Mind? Stress Testing Social Reasoning in Large Language Modelscited by 161
  5. 2021Can Machines Learn Morality? The Delphi Experimentcited by 140
  6. 2021The Benchmark Lotterycited by 133
  7. 2020The Effect of Natural Distribution Shift on Question Answering Modelscited by 104
  8. 2021Reduced, Reused and Recycled: The Life of a Dataset in Machine Learning Researchcited by 100
  9. 2018The Measure and Mismeasure of Fairnesscited by 94
  10. 2020From ImageNet to Image Classification: Contextualizing Progress on Benchmarkscited by 90
  11. 2015Generalization in Adaptive Data Analysis and Holdout Reusecited by 76
  12. 2015The Ladder: A Reliable Leaderboard for Machine Learning Competitionscited by 67
  13. 2019Cold Case: The Lost MNIST Digitscited by 61
  14. 2025Measurement to Meaning: A Validity-Centered Framework for AI Evaluationcited by 54
  15. 2020Targeting the Benchmark: On Methodology in Current Natural Language Processing Researchcited by 44
  16. 2025Large Language Model Psychometrics: A Systematic Review of Evaluation, Validation, and Enhancementcited by 41
  17. 2023Difficult Lessons on Social Prediction from Wisconsin Public Schoolscited by 35
  18. 2023Using Sequences of Life-events to Predict Human Livescited by 27
  19. 2024Inherent Trade-Offs between Diversity and Stability in Multi-Task Benchmarkscited by 26
  20. 2023Evaluating General-Purpose AI with Psychometricscited by 23
  21. 2022Scientific Inference With Interpretable Machine Learning: Analyzing Models to Learn About Real-World Phenomenacited by 21
  22. 2024A Philosophical Introduction to Language Models - Part II: The Way Forwardcited by 17
  23. 2022The no-free-lunch theorems of supervised learningcited by 16
  24. 2020Large-scale benchmark study of survival prediction methods using multi-omics datacited by 14
  25. 2024ImageNot: A contrast with ImageNet preserves model rankingscited by 14
  26. 2024From the Fair Distribution of Predictions to the Fair Distribution of Social Goods: Evaluating the Impact of Fair Machine Learning on Long-Term Unemploymentcited by 12
  27. 2022Making Intelligence: Ethical Values in IQ and ML Benchmarkscited by 10
  28. 2022Deep Learning Opacity in Scientific Discoverycited by 9
  29. 2023What Makes ImageNet Look Unlike LAIONcited by 9
  30. 2020Decoding machine learning benchmarkscited by 8
  31. 2025Position: Stop Evaluating AI with Human Tests, Develop Principled, AI-specific Tests insteadcited by 7
  32. 2021Fairness in Algorithmic Profiling: A German Case Studycited by 6
  33. 2023Statistical learning theory and Occam's razor: The core argumentcited by 5
  34. 2024The Costs of Pretending That There Are Data-Generating Probability Distributions in the Social Worldcited by 4
  35. 2024Combining the Strengths of Dutch Survey and Register Data in a Data Challenge to Predict Fertility (PreFer)cited by 3

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2026: 12.

12 papers.

  1. 2026Establishing Construct Validity in LLM Capability Benchmarks Requires Nomological Networkscited by 6
  2. 2026The COTe score: A decomposable framework for evaluating Document Layout Analysis modelscited by 1
  3. 2026Harbor Adapters and Harbor-Index: Infrastructure and a Curated Meta-Dataset for Large-Scale Agentic Evaluationcited by 1
  4. 2026Falsifying Discriminant Validity of Predictive Algorithmscited by 0
  5. 2026Questionnaire Responses Do not Capture the Safety of AI Agentscited by 0
  6. 2026From Controlled to the Wild: Evaluation of Pentesting Agents for the Real-Worldcited by 0
  7. 2026How Hard is it to Rig a Benchmark? A Social Choice Analysis of Leaderboard Robustnesscited by 0
  8. 2026Validity Threats for Foundation Model Researchcited by 0
  9. 2026Exposure is not manifestation: measurement target and output resolution jointly determine which behavioural-faithfulness evaluator winscited by 0
  10. 2026When benchmark inferences do not compose: Projectibility in AI evaluationcited by 0
  11. 2026Clean Engineering, Unstable Measurement: A Preregistered Reliability Failure of Black-Box LLM Observers on Shared Endpointscited by 0
  12. 2026When Evidence Changes the Subject: Subject-Typed Claim Licensing for Learned Routingcited by 0