Frontier AI Forecasting Has a Measurement Problem: An Audit of Progress Evidence
Not on the map. Why?
Abstract (arXiv)
Quantitative forecasts of frontier artificial intelligence often connect dated targets to trends in benchmark scores, training compute, release time, or expert belief. This paper audits whether the public measurement record supports those connections before another trend is fitted. I construct a frozen, event-centric record through 12 August 2026 with 62 selected systems, 12 versioned benchmarks, seven capability or impact criteria, 144 graded events, 27 source records, and 408 typed relations. The record is an audit sample, not a census. Only seven systems jointly observe estimated training compute and a METR 50 percent task horizon. Training compute is absent for 19 of 27 closed systems, including every selected closed release from 2026, while none of the 35 open-weight systems has a METR horizon observation. Benchmark succession creates a second break: a seven-system link from METR Time Horizon 1.0 to 1.1 has a log-scale slope of 1.206 (95 percent CI 1.021 to 1.390), whereas a six-system MMLU to MMLU-Pro comparison appears shift-like under logit and probit links but not under linear or logarithmic links. The observed bridges have about 80 percent power only for slope departures near 25 percent. Provenance is concentrated: 52 of 71 substantive quantitative events, or 73.2 percent, come from one measurement programme, and 76.1 percent are laboratory releases. A review of 56 methodological and empirical sources identifies 16 complementary measurement directions spanning resources, inference budgets, reliability, agentic work, safety, human preference, field outcomes, and forecast backtesting. No direction supplies a replacement scalar. The result is not that frontier AI forecasting is impossible, but that a defensible dated forecast is a claim about a versioned measurement system with explicit joins, protocols, links, and source dependence, not merely a fitted curve or calendar date.
Key facts. No citing papers in the map yet (data through 7 Oct 2026). Builds on 38 of its 61 references that are in the map. First posted on arXiv 14 Aug 2026.
Builds on
38 of 61 references.
References · 38
- 2024Scaling LLM Test-Time Compute Optimally can be More Effective than Scaling Model Parameterscited by 1,928
- 2024MMLU-Pro: A More Robust and Challenging Multi-Task Language Understanding Benchmarkcited by 1,758
- 2023Pythia: A Suite for Analyzing Large Language Models Across Training and Scalingcited by 1,734
- 2018Datasheets for Datasetscited by 1,371
- 2024HarmBench: A Standardized Evaluation Framework for Automated Red Teaming and Robust Refusalcited by 1,330
- 2024Chatbot Arena: An Open Platform for Evaluating LLMs by Human Preferencecited by 1,151
- 2018Model Cards for Model Reportingcited by 1,068
- 2025Humanity's Last Examcited by 467
- 2024tinyBenchmarks: evaluating LLMs with fewer examplescited by 295
- 2025PaperBench: Evaluating AI's Ability to Replicate AI Researchcited by 286
- 2023Model evaluation for extreme riskscited by 167
- 2024RE-Bench: Evaluating frontier AI R&D capabilities of language model agents against human expertscited by 153
26 more
- 2020With Little Power Comes Great Responsibilitycited by 138
- 2023Levels of AGI for Operationalizing Progress on the Path to AGIcited by 94
- 2025Measuring the Impact of Early-2025 AI on Experienced Open-Source Developer Productivitycited by 93
- 2024Thousands of AI Authors on the Future of AIcited by 48
- 2017When Will AI Exceed Human Performance? Evidence from AI Expertscited by 41
- 2023Evaluating Evaluation Metrics: A Framework for Analyzing NLG Evaluation Metrics using Measurement Theorycited by 41
- 2024MMLU-CF: A Contamination-free Multi-task Language Understanding Benchmarkcited by 41
- 2023Position: AI Evaluation Should Learn from How We Test Humanscited by 36
- 2025Medical Large Language Model Benchmarks Should Prioritize Construct Validitycited by 36
- 2024Algorithmic progress in language modelscited by 31
- 2025A Rosetta Stone for AI Benchmarkscited by 29
- 2026When AI Benchmarks Plateau: A Systematic Study of Benchmark Saturationcited by 29
- 2024Do These LLM Benchmarks Agree? Fixing Benchmark Evaluation with BenchBenchcited by 25
- 2025Persistent Instability in LLM's Personality Measurements: Effects of Scale, Reasoning, and Conversation Historycited by 25
- 2025Adaptive Testing for LLM Evaluation: A Psychometric Alternative to Static Benchmarkscited by 25
- 2025Dynamic Benchmarking of Reasoning Capabilities in Code Large Language Models Under Data Contaminationcited by 22
- 2026HLE-Verified: A Systematic Verification and Structured Revision of Humanity's Last Examcited by 11
- 2026Capabilities Ain't All You Need: Measuring Propensities in AIcited by 6
- 2026When the Judge Changes, So Does the Measurement: Auditing LLM-as-Judge Reliabilitycited by 6
- 2025Forecasting Frontier Language Model Agent Capabilitiescited by 5
- 2026How Inference Compute Shapes Frontier LLM Evaluationcited by 5
- 2024Improving the Validity and Practical Usefulness of AI/ML Evaluations Using an Estimands Frameworkcited by 4
- 2025Garbage In, Reasoning Out? Why Benchmark Scores are Unreliable and What to Do About Itcited by 4
- 2026Can We Trust Item Response Theory for AI Evaluation?cited by 3
- 2026Correcting Human Labels for Rater Effects in AI Evaluation: An Item Response Theory Approachcited by 1
- 2026Item Response Theory for AI Safetycited by 1
Built on by
Not cited yet.
Similar, not cited
Close in wording, no citation either way
- 2026Bayesian Inference and Decision Audits for Public Archives of Frontier AI Evaluations
- 2026Scientific reasoning does not reliably translate into scientific forecasting in frontier AI
- 2026Frontier Lag: A Bibliometric Audit of Capability Misrepresentation in Academic AI Evaluation
- 2025Bench to the Future: A Pastcasting Benchmark for Forecasting Agents
- 2026Tracing the Evidence Behind Zero-Shot Time-Series Forecasting: A Source-First Taxonomy and Audit Framework
- 2026Thomson: Continual Learning of Frontier Models for SovereignAI
- 2026The Growing Pains of Frontier Models: When Leaderboards Stop Separating and What to Measure Next
- 2026FrontierFinance: A Challenging Benchmark for Measuring Frontier Intelligence of Finance Agents