Bayesian Inference and Decision Audits for Public Archives of Frontier AI Evaluations
Not on the map. Why?
Abstract (arXiv)
Public AI evaluations are often read as terminal leaderboards, yet the underlying evidence is a selective time series shaped by reporting rules, benchmark revisions, and missingness. Repeated public archives for LiveBench and Open LLM Leaderboard v2 serve as the primary longitudinal record; LMArena provides a preference stress test; and GAIA and tau-bench contribute limited agentic pilots. Together, these archives instantiate a Bayesian inference problem: under a fixed reporting convention, one constructed terminal-only example over 1,000 systems is compatible with two pre-terminal histories, yielding times of 23.03 or 75.13 to reach within 0.05 of the ceiling under the same terminal-tail model. In synthetic posterior comparisons, action-facing diagnostics differ across observation regimes. The candidate selection-aware frontier model fails synthetic recovery, objective-archive prediction, preference transfer, and uncertainty calibration; correspondingly, fixed audit gates reject its stronger claims. An archive-and-adjudication protocol reconstructs public evaluation histories, isolates a verified timing boundary, and falsifies unsupported frontier claims.
Key facts. No citing papers in the map yet (data through 7 Oct 2026). Builds on 23 of its 39 references that are in the map. First posted on arXiv 15 Jun 2026.
Builds on
23 of 39 references.
References · 23
- 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,393
- 2020Scaling Laws for Neural Language Modelscited by 6,411
- 2023SWE-bench: Can Language Models Resolve Real-World GitHub Issues?cited by 3,193
- 2022Beyond the Imitation Game: Quantifying and extrapolating the capabilities of language modelscited by 1,970
- 2023WebArena: A Realistic Web Environment for Building Autonomous Agentscited by 1,908
- 2023ToolLLM: Facilitating Large Language Models to Master 16000+ Real-world APIscited by 1,880
- 2024τ-bench: A Benchmark for Tool-Agent-User Interaction in Real-World Domainscited by 1,280
- 2023AgentBench: Evaluating LLMs as Agentscited by 1,211
- 2024Chatbot Arena: An Open Platform for Evaluating LLMs by Human Preferencecited by 1,151
- 2023GAIA: a benchmark for General AI Assistantscited by 1,084
- 2023Are Emergent Abilities of Large Language Models a Mirage?cited by 515
- 2023NLP Evaluation in trouble: On the Need to Measure LLM Data Contamination for each Benchmarkcited by 339
11 more
- 2024LiveBench: A Challenging, Contamination-Limited LLM Benchmarkcited by 242
- 2021What Will it Take to Fix Benchmarking in Natural Language Understanding?cited by 200
- 2019Show Your Work: Improved Reporting of Experimental Resultscited by 164
- 2020With Little Power Comes Great Responsibilitycited by 138
- 2024When Benchmarks are Targets: Revealing the Sensitivity of Large Language Model Leaderboardscited by 134
- 2015Generalization in Adaptive Data Analysis and Holdout Reusecited by 76
- 2015The Ladder: A Reliable Leaderboard for Machine Learning Competitionscited by 67
- 2021Comparing Test Sets with Item Response Theorycited by 55
- 2021ExplainaBoard: An Explainable Leaderboard for NLPcited by 50
- 2024A Flexible Defense Against the Winner's Cursecited by 6
- 2024Testing for sufficient follow-up in survival data with a cure fractioncited by 1
Built on by
Not cited yet.
Similar, not cited
Close in wording, no citation either way
- 2026Frontier AI Forecasting Has a Measurement Problem: An Audit of Progress Evidence
- 2026HindsightBench: A Black-Box Behavioral Audit Protocol for Parametric Hindsight in Time-Indexed LLM Decision Tasks
- 2026Test-Time Scaling in Reasoning LLMs: Inference Regimes, Evaluation, and Reproducibility
- 2026The Growing Pains of Frontier Models: When Leaderboards Stop Separating and What to Measure Next
- 2024No Free Delivery Service: Epistemic limits of passive data collection in complex social systems
- 2026From Trajectories to Evidence: Auditable Experimental Records for Industrial Research Agents
- 2025HiBayES: A Hierarchical Bayesian Modeling Framework for AI Evaluation Statistics
- 2026When Validation Stops Learning: Auditing Update Admission for Continual Embodied Agents