mlsanityData to 7 Oct 2026

Benchmarks as Microscopes: A Call for Model Metrology

arXiv 2407.16711 ↗First posted 22 Jul 2024cs.SE cs.CLcited by 33

Not on the map. Why?

Abstract (arXiv)

Modern language models (LMs) pose a new challenge in capability assessment. Static benchmarks inevitably saturate without providing confidence in the deployment tolerances of LM-based systems, but developers nonetheless claim that their models have generalized traits such as reasoning or open-domain language understanding based on these flawed metrics. The science and practice of LMs requires a new approach to benchmarking which measures specific capabilities with dynamic assessments. To be confident in our metrics, we need a new discipline of model metrology -- one which focuses on how to generate benchmarks that predict performance under deployment. Motivated by our evaluation criteria, we outline how building a community of model metrology practitioners -- one focused on building tools and studying how to measure system capabilities -- is the best way to meet these needs to and add clarity to the AI discussion.

Key facts. Cited by 33 papers in the map (data through 7 Oct 2026). Builds on 83 of its 137 references that are in the map. First posted on arXiv 22 Jul 2024.

Builds on

83 of 137 references.

References · 83

  1. 2023GPT-4 Technical Reportcited by 18,132
  2. 2021Training Verifiers to Solve Math Word Problemscited by 9,743
  3. 2014ImageNet Large Scale Visual Recognition Challengecited by 8,441
  4. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,393
  5. 2021Evaluating Large Language Models Trained on Codecited by 8,053
  6. 2020Measuring Massive Multitask Language Understandingcited by 7,768
  7. 2021On the Opportunities and Risks of Foundation Modelscited by 3,961
  8. 2022Training Compute-Optimal Large Language Modelscited by 2,927
  9. 2023Sparks of Artificial General Intelligence: Early experiments with GPT-4cited by 2,579
  10. 2024LiveCodeBench: Holistic and Contamination Free Evaluation of Large Language Models for Codecited by 1,906
  11. 2015A large annotated corpus for learning natural language inferencecited by 1,837
  12. 2019SuperGLUE: A Stickier Benchmark for General-Purpose Language Understanding Systemscited by 1,718
38 more
  1. 2023MathVista: Evaluating Mathematical Reasoning of Foundation Models in Visual Contextscited by 1,582
  2. 2021BEIR: A Heterogenous Benchmark for Zero-shot Evaluation of Information Retrieval Modelscited by 1,225
  3. 2024Chatbot Arena: An Open Platform for Evaluating LLMs by Human Preferencecited by 1,151
  4. 2023Large Language Models are not Fair Evaluatorscited by 1,050
  5. 2020Beyond Accuracy: Behavioral Testing of NLP models with CheckListcited by 979
  6. 2024Are We on the Right Way for Evaluating Large Vision-Language Models?cited by 893
  7. 2019Right for the Wrong Reasons: Diagnosing Syntactic Heuristics in Natural Language Inferencecited by 864
  8. 2023Can Large Language Models Be an Alternative to Human Evaluations?cited by 729
  9. 2023ChatEval: Towards Better LLM-based Evaluators through Multi-Agent Debatecited by 724
  10. 2023Are Emergent Abilities of Large Language Models a Mirage?cited by 515
  11. 2022Large Language Models Struggle to Learn Long-Tail Knowledgecited by 452
  12. 2023Large Language Models Are State-of-the-Art Evaluators of Translation Qualitycited by 416
  13. 2023TIFA: Accurate and Interpretable Text-to-Image Faithfulness Evaluation with Question Answeringcited by 416
  14. 2023The Reversal Curse: LLMs trained on "A is B" fail to learn "B is A"cited by 399
  15. 2021QA-GNN: Reasoning with Language Models and Knowledge Graphs for Question Answeringcited by 386
  16. 2021AI and the Everything in the Whole Wide World Benchmarkcited by 340
  17. 2020Are we done with ImageNet?cited by 329
  18. 2019Measurement and Fairnesscited by 299
  19. 2024tinyBenchmarks: evaluating LLMs with fewer examplescited by 295
  20. 2018Hypothesis Only Baselines in Natural Language Inferencecited by 291
  21. 2023Large Language Models as Commonsense Knowledge for Large-Scale Task Planningcited by 231
  22. 2021Mind the Gap: Assessing Temporal Generalization in Neural Language Modelscited by 226
  23. 2023Inverse Scaling: When Bigger Isn't Bettercited by 205
  24. 2023OpenAGI: When LLM Meets Domain Expertscited by 196
  25. 2023Benchmarking Foundation Models with Language-Model-as-an-Examinercited by 172
  26. 2023Solving Challenging Math Word Problems Using GPT-4 Code Interpreter with Code-based Self-Verificationcited by 166
  27. 2023Davidsonian Scene Graph: Improving Reliability in Fine-grained Evaluation for Text-to-Image Generationcited by 159
  28. 2022The Debate Over Understanding in AI's Large Language Modelscited by 142
  29. 2024On the Self-Verification Limitations of Large Language Models on Reasoning and Planning Taskscited by 137
  30. 2024When Benchmarks are Targets: Revealing the Sensitivity of Large Language Model Leaderboardscited by 134
  31. 2023What's In My Big Data?cited by 129
  32. 2022Toward Transparent AI: A Survey on Interpreting the Inner Structures of Deep Neural Networkscited by 126
  33. 2021Towards artificial general intelligence via a multimodal foundation modelcited by 112
  34. 2023M3Exam: A Multilingual, Multimodal, Multilevel Benchmark for Examining Large Language Modelscited by 109
  35. 2021Reduced, Reused and Recycled: The Life of a Dataset in Machine Learning Researchcited by 100
  36. 2023The Generative AI Paradox: "What It Can Create, It May Not Understand"cited by 97
  37. 2023CoMPosT: Characterizing and Evaluating Caricature in LLM Simulationscited by 94
  38. 2023Levels of AGI for Operationalizing Progress on the Path to AGIcited by 94

50 most cited of 83 All 83 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 4; 2025: 17; 2026: 12.

33 papers.

Sort:

By area:

  1. 2024Do Vision-Language Models Represent Space and How? Evaluating Spatial Frame of Reference Under Ambiguitiescited by 58
  2. 2025Which of These Best Describes Multiple Choice Evaluation with LLMs? A) Forced B) Flawed C) Fixable D) All of the Abovecited by 51
  3. 2025Toward an Evaluation Science for Generative AI Systemscited by 42
  4. 2025Fluid Language Model Benchmarkingcited by 30
  5. 2026When AI Benchmarks Plateau: A Systematic Study of Benchmark Saturationcited by 29
  6. 2025EvalTree: Profiling Language Model Weaknesses via Hierarchical Capability Treescited by 26
  7. 2025Do Vision-Language Models Have Internal World Models? Towards an Atomic Evaluationcited by 23
  8. 2025IssueBench: Millions of Realistic Prompts for Measuring Issue Bias in LLM Writing Assistancecited by 22
  9. 2024Improving Model Evaluation using SMART Filtering of Benchmark Datasetscited by 20
  10. 2024ONEBench to Test Them All: Sample-Level Benchmarking Over Open-Ended Capabilitiescited by 13
  11. 2025Culture is Everywhere: A Call for Intentionally Cultural Evaluationcited by 10
  12. 2025Vision-Language Models Are Not Pragmatically Competent in Referring Expression Generationcited by 8
21 more
  1. 2025How Reliable is Language Model Micro-Benchmarking?cited by 8
  2. 2025A Good Plan is Hard to Find: Aligning Models with Preferences is Misaligned with What Helps Userscited by 6
  3. 2026Emergence WebVoyager: Toward Consistent and Transparent Evaluation of (Web) Agents in The Wildcited by 5
  4. 2025Toward a Theory of Generalizability in LLM Mechanistic Interpretability Researchcited by 4
  5. 2025Societal Impacts Research Requires Benchmarks for Creative Composition Taskscited by 3
  6. 2025BenchHub: A Unified Benchmark Suite for Holistic and Customizable LLM Evaluationcited by 3
  7. 2026BenchMarker: An Education-Inspired Toolkit for Highlighting Flaws in Multiple-Choice Benchmarkscited by 3
  8. 2024Can Models Help Us Create Better Models? Evaluating LLMs as Data Scientistscited by 2
  9. 2026Language Models Don't Know What You Want: Evaluating Personalization in Deep Research Needs Real Userscited by 2
  10. 2025Evaluating Large Language Models for Fair and Reliable Organ Allocationcited by 1
  11. 2025Can They Dixit? Yes they Can! Dixit as a Playground for Multimodal Language Model Capabilitiescited by 1
  12. 2025The Differential Meaning of Models: A Framework for Analyzing the Structural Consequences of Semantic Modeling Decisionscited by 0
  13. 2025Towards Real-World Validity in Generative AI Benchmarks: Understanding and Designing Domain-Centered Evaluations for Journalism Practitionerscited by 0
  14. 2026Aligning Language Model Benchmarks with Pairwise Preferencescited by 0
  15. 2026Correcting test set contamination by spiking the training datacited by 0
  16. 2026Developmental Trajectories of Situation Modeling and Mentalizing in Transformer Language Modelscited by 0
  17. 2026Optimization Is Not All You Needcited by 0
  18. 2026Check The Scoreboard: An Analysis of Scoring Schemes on Multiple-Choice Evaluationcited by 0
  19. 2026API Benchmark Scores Do Not Reliably Transfer to Chatbot Interfacescited by 0
  20. 2026WordPolo: Evaluating Language Models Through Iterative Semantic Feedbackcited by 0
  21. 2026WitnessGym: Benchmarking Coding Agents on the Construction of Bug Witnessescited by 0