mlsanityData to 7 Oct 2026

Bayesian Inference and Decision Audits for Public Archives of Frontier AI Evaluations

arXiv 2606.17005 ↗First posted 15 Jun 2026cs.AI stat.MEcited by 0

Not on the map. Why?

Abstract (arXiv)

Public AI evaluations are often read as terminal leaderboards, yet the underlying evidence is a selective time series shaped by reporting rules, benchmark revisions, and missingness. Repeated public archives for LiveBench and Open LLM Leaderboard v2 serve as the primary longitudinal record; LMArena provides a preference stress test; and GAIA and tau-bench contribute limited agentic pilots. Together, these archives instantiate a Bayesian inference problem: under a fixed reporting convention, one constructed terminal-only example over 1,000 systems is compatible with two pre-terminal histories, yielding times of 23.03 or 75.13 to reach within 0.05 of the ceiling under the same terminal-tail model. In synthetic posterior comparisons, action-facing diagnostics differ across observation regimes. The candidate selection-aware frontier model fails synthetic recovery, objective-archive prediction, preference transfer, and uncertainty calibration; correspondingly, fixed audit gates reject its stronger claims. An archive-and-adjudication protocol reconstructs public evaluation histories, isolates a verified timing boundary, and falsifies unsupported frontier claims.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Builds on 23 of its 39 references that are in the map. First posted on arXiv 15 Jun 2026.

Builds on

23 of 39 references.

References · 23

  1. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,393
  2. 2020Scaling Laws for Neural Language Modelscited by 6,411
  3. 2023SWE-bench: Can Language Models Resolve Real-World GitHub Issues?cited by 3,193
  4. 2022Beyond the Imitation Game: Quantifying and extrapolating the capabilities of language modelscited by 1,970
  5. 2023WebArena: A Realistic Web Environment for Building Autonomous Agentscited by 1,908
  6. 2023ToolLLM: Facilitating Large Language Models to Master 16000+ Real-world APIscited by 1,880
  7. 2024τ-bench: A Benchmark for Tool-Agent-User Interaction in Real-World Domainscited by 1,280
  8. 2023AgentBench: Evaluating LLMs as Agentscited by 1,211
  9. 2024Chatbot Arena: An Open Platform for Evaluating LLMs by Human Preferencecited by 1,151
  10. 2023GAIA: a benchmark for General AI Assistantscited by 1,084
  11. 2023Are Emergent Abilities of Large Language Models a Mirage?cited by 515
  12. 2023NLP Evaluation in trouble: On the Need to Measure LLM Data Contamination for each Benchmarkcited by 339
11 more
  1. 2024LiveBench: A Challenging, Contamination-Limited LLM Benchmarkcited by 242
  2. 2021What Will it Take to Fix Benchmarking in Natural Language Understanding?cited by 200
  3. 2019Show Your Work: Improved Reporting of Experimental Resultscited by 164
  4. 2020With Little Power Comes Great Responsibilitycited by 138
  5. 2024When Benchmarks are Targets: Revealing the Sensitivity of Large Language Model Leaderboardscited by 134
  6. 2015Generalization in Adaptive Data Analysis and Holdout Reusecited by 76
  7. 2015The Ladder: A Reliable Leaderboard for Machine Learning Competitionscited by 67
  8. 2021Comparing Test Sets with Item Response Theorycited by 55
  9. 2021ExplainaBoard: An Explainable Leaderboard for NLPcited by 50
  10. 2024A Flexible Defense Against the Winner's Cursecited by 6
  11. 2024Testing for sufficient follow-up in survival data with a cure fractioncited by 1

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2026Frontier AI Forecasting Has a Measurement Problem: An Audit of Progress Evidence
  2. 2026HindsightBench: A Black-Box Behavioral Audit Protocol for Parametric Hindsight in Time-Indexed LLM Decision Tasks
  3. 2026Test-Time Scaling in Reasoning LLMs: Inference Regimes, Evaluation, and Reproducibility
  4. 2026The Growing Pains of Frontier Models: When Leaderboards Stop Separating and What to Measure Next
  5. 2024No Free Delivery Service: Epistemic limits of passive data collection in complex social systems
  6. 2026From Trajectories to Evidence: Auditable Experimental Records for Industrial Research Agents
  7. 2025HiBayES: A Hierarchical Bayesian Modeling Framework for AI Evaluation Statistics
  8. 2026When Validation Stops Learning: Auditing Update Admission for Continual Embodied Agents