mlsanityData to 14 Oct 2026

What Does an Evaluation License? A Commit-Bound Census of Claim Replay in Inspect Evals

arXiv 2608.19269 ↗First posted 18 Aug 2026cs.SE cs.AIcited by 0

Not on the map. Why?

Abstract (arXiv)

Benchmarks can run without determining what their results license. We freeze a large evaluation collection and attempt to replay its historical claims. Most units stop because the evidence required for replay is not bound. Where replay is possible, different claims remain stable at different resolutions. We make this otherwise implicit inference step explicit and executable.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 14 of its 32 references that are in the map. First posted on arXiv 18 Aug 2026.

Builds on

14 of 32 references.

References · 14

  1. 2020Beyond Accuracy: Behavioral Testing of NLP models with CheckListcited by 1,006
  2. 2020Underspecification Presents Challenges for Credibility in Modern Machine Learningcited by 496
  3. 2021Dynabench: Rethinking Benchmarking in NLPcited by 459
  4. 2021AI and the Everything in the Whole Wide World Benchmarkcited by 349
  5. 2019Measurement and Fairnesscited by 308
  6. 2021What Will it Take to Fix Benchmarking in Natural Language Understanding?cited by 203
  7. 2021Accounting for Variance in Machine Learning Benchmarkscited by 201
  8. 2021The Benchmark Lotterycited by 136
  9. 2025Measuring what Matters: Construct Validity in Large Language Model Benchmarkscited by 119
  10. 2025Measurement to Meaning: A Validity-Centered Framework for AI Evaluationcited by 57
  11. 2022AMLB: an AutoML Benchmarkcited by 55
  12. 2023Weak Supervision Performance Evaluation via Partial Identificationcited by 4
2 more
  1. 2026Deployment-complete benchmarkingcited by 1
  2. 2026Auditing Frame-Level AUC in Weakly Supervised Video Anomaly Detection: Granularity, Resolution, and Scene Biascited by 1

Built on by

Not cited yet.