mlsanityData to 7 Oct 2026

BenchBrowser: Retrieving Evidence for Evaluating Benchmark Validity

arXiv 2603.18019 ↗First posted 25 Feb 2026cs.CL cs.AI cs.SEcited by 0

Not on the map. Why?

Abstract (arXiv)

Do language model benchmarks actually measure what practitioners intend them to ? High-level metadata is too coarse to convey the granular reality of benchmarks: a "poetry" benchmark may never test for haikus, while "instruction-following" benchmarks will often test for an arbitrary mix of skills. This opacity makes verifying alignment with practitioner goals a laborious process, risking an illusion of competence even when models fail on untested facets of user interests. We introduce BenchBrowser, a retriever that surfaces evaluation items relevant to natural language use cases over 20 benchmark suites. Validated by a human study confirming high retrieval precision, BenchBrowser generates evidence to help practitioners diagnose low content validity (narrow coverage of a capability's facets) and low convergent validity (lack of stable rankings when measuring the same capability). BenchBrowser, thus, helps quantify a critical gap between practitioner intent and what benchmarks actually test.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Builds on 65 of its 91 references that are in the map. First posted on arXiv 25 Feb 2026.

Builds on

65 of 91 references.

References · 65

  1. 2024The Llama 3 Herd of Modelscited by 14,192
  2. 2021Training Verifiers to Solve Math Word Problemscited by 9,743
  3. 2021Evaluating Large Language Models Trained on Codecited by 8,053
  4. 2020Measuring Massive Multitask Language Understandingcited by 7,768
  5. 2024Qwen2.5 Technical Reportcited by 4,713
  6. 2018Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challengecited by 4,710
  7. 2019HellaSwag: Can a Machine Really Finish Your Sentence?cited by 4,018
  8. 2021TruthfulQA: Measuring How Models Mimic Human Falsehoodscited by 2,919
  9. 2017TriviaQA: A Large Scale Distantly Supervised Challenge Dataset for Reading Comprehensioncited by 2,895
  10. 2018Can a Suit of Armor Conduct Electricity? A New Dataset for Open Book Question Answeringcited by 2,114
  11. 2022Beyond the Imitation Game: Quantifying and extrapolating the capabilities of language modelscited by 1,970
  12. 2022Challenging BIG-Bench Tasks and Whether Chain-of-Thought Can Solve Themcited by 1,767
38 more
  1. 2019PubMedQA: A Dataset for Biomedical Research Question Answeringcited by 1,271
  2. 2023Instruction-Following Evaluation for Large Language Modelscited by 1,143
  3. 2024Tulu 3: Pushing Frontiers in Open Language Model Post-Trainingcited by 988
  4. 2024Length-Controlled AlpacaEval: A Simple Way to Debias Automatic Evaluatorscited by 865
  5. 2017Crowdsourcing Multiple Choice Science Questionscited by 742
  6. 2019MathQA: Towards Interpretable Math Word Problem Solving with Operation-Based Formalismscited by 653
  7. 2024The WMDP Benchmark: Measuring and Reducing Malicious Use With Unlearningcited by 488
  8. 2021A Dataset of Information-Seeking Questions and Answers Anchored in Research Paperscited by 475
  9. 2024The Faiss librarycited by 370
  10. 2023Query Rewriting for Retrieval-Augmented Large Language Modelscited by 211
  11. 2024Benchmarking Complex Instruction-Following with Multiple Constraints Compositioncited by 147
  12. 2020Moral Stories: Situated Reasoning about Norms, Intents, Actions, and their Consequencescited by 133
  13. 2024InFoBench: Evaluating Instruction Following Ability in Large Language Modelscited by 128
  14. 2024BetterBench: Assessing AI Benchmarks, Uncovering Issues, and Establishing Best Practicescited by 112
  15. 2025Measuring what Matters: Construct Validity in Large Language Model Benchmarkscited by 102
  16. 2019HEAD-QA: A Healthcare Dataset for Complex Reasoningcited by 100
  17. 2024Optimization Methods for Personalizing Large Language Models through Retrieval Augmentationcited by 98
  18. 2024Clio: Privacy-Preserving Insights into Real-World AI Usecited by 66
  19. 2023A Multi-Task Embedder For Retrieval Augmented LLMscited by 63
  20. 2025Position: Evaluating Generative AI Systems Is a Social Science Measurement Challengecited by 61
  21. 2025Can We Trust AI Benchmarks? An Interdisciplinary Review of Current Issues in AI Evaluationcited by 57
  22. 2025Command A: An Enterprise-Ready Large Language Modelcited by 57
  23. 2025Measurement to Meaning: A Validity-Centered Framework for AI Evaluationcited by 54
  24. 2023Bias Testing and Mitigation in LLM-based Code Generationcited by 53
  25. 2023From Sparse to Dense: GPT-4 Summarization with Chain of Density Promptingcited by 52
  26. 2024Cheating Automatic LLM Benchmarks: Null Models Achieve High Win Ratescited by 42
  27. 2025Lost in Benchmarks? Rethinking Large Language Model Benchmarking with Item Response Theorycited by 42
  28. 2025Large Language Model Psychometrics: A Systematic Review of Evaluation, Validation, and Enhancementcited by 41
  29. 2024Can Large Language Models be Trusted for Evaluation? Scalable Meta-Evaluation of LLMs as Evaluators via Agent Debatecited by 40
  30. 2024AutoBencher: Towards Declarative Benchmark Constructioncited by 40
  31. 2025Medical Large Language Model Benchmarks Should Prioritize Construct Validitycited by 36
  32. 2024MM-Eval: A Multilingual Meta-Evaluation Benchmark for LLM-as-a-Judge and Reward Modelscited by 34
  33. 2025Fluid Language Model Benchmarkingcited by 30
  34. 2024ECBD: Evidence-Centered Benchmark Design for NLPcited by 27
  35. 2025EvalTree: Profiling Language Model Weaknesses via Hierarchical Capability Treescited by 26
  36. 2025Finding Flawed Fictions: Evaluating Complex Reasoning in Language Models via Plot Hole Detectioncited by 24
  37. 2025YourBench: Easy Custom Evaluation Sets for Everyonecited by 20
  38. 2025Redundancy Principles for MLLMs Benchmarkscited by 18

50 most cited of 65 All 65 (JSON)

Built on by

Not cited yet.