mlsanityData to 14 Oct 2026

A Rosetta Stone for AI Benchmarks

arXiv 2512.00193 ↗First posted 28 Nov 2025cs.AIcited by 36

Not on the map. Why?

Abstract (arXiv)

Most AI benchmarks saturate within years or even months after they are introduced, making it hard to study long-run trends in AI capabilities. To address this challenge, we build a statistical framework that stitches benchmarks together, putting model capabilities and benchmark difficulties on a single numerical scale. This acts as a "Rosetta Stone", allowing us to compare models across a wide range of abilities and time, even if they are not evaluated on the same benchmarks. Moreover, this works without assuming how capabilities evolve across time or with training compute. We demonstrate three applications of this framework. First, we use it to measure the speed of AI progress over time, and to forecast future AI capabilities. Second, we estimate the rate of improvements in algorithmic efficiency, finding estimates that are higher, but broadly consistent with prior work. Finally, we find that our approach can be used to detect rapid accelerations in AI progress.

Key facts. Cited by 36 papers in the map (data through 14 Oct 2026). Builds on 40 of its 64 references that are in the map. First posted on arXiv 28 Nov 2025.

Builds on

40 of 64 references.

References · 40

  1. 2021Training Verifiers to Solve Math Word Problemscited by 10,109
  2. 2020Measuring Massive Multitask Language Understandingcited by 7,963
  3. 2021Measuring Mathematical Problem Solving With the MATH Datasetcited by 5,371
  4. 2018Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challengecited by 4,886
  5. 2019HellaSwag: Can a Machine Really Finish Your Sentence?cited by 4,100
  6. 2017TriviaQA: A Large Scale Distantly Supervised Challenge Dataset for Reading Comprehensioncited by 2,955
  7. 2019PIQA: Reasoning about Physical Commonsense in Natural Languagecited by 2,883
  8. 2023GPQA: A Graduate-Level Google-Proof Q&A Benchmarkcited by 2,879
  9. 2018Can a Suit of Armor Conduct Electricity? A New Dataset for Open Book Question Answeringcited by 2,164
  10. 2022Learn to Explain: Multimodal Reasoning via Thought Chains for Science Question Answeringcited by 1,919
  11. 2022Challenging BIG-Bench Tasks and Whether Chain-of-Thought Can Solve Themcited by 1,807
  12. 2019SuperGLUE: A Stickier Benchmark for General-Purpose Language Understanding Systemscited by 1,729
28 more
  1. 2024Video-MME: The First-Ever Comprehensive Evaluation Benchmark of Multi-modal LLMs in Video Analysiscited by 1,413
  2. 2024OSWorld: Benchmarking Multimodal Agents for Open-Ended Tasks in Real Computer Environmentscited by 1,195
  3. 2016The LAMBADA dataset: Word prediction requiring a broad discourse contextcited by 992
  4. 2019Adversarial NLI: A New Benchmark for Natural Language Understandingcited by 807
  5. 2024TheAgentCompany: Benchmarking LLM Agents on Consequential Real World Taskscited by 326
  6. 2024tinyBenchmarks: evaluating LLMs with fewer examplescited by 310
  7. 2024LiveBench: A Challenging, Contamination-Limited LLM Benchmarkcited by 262
  8. 2025DeepResearch Bench: A Comprehensive Benchmark for Deep Research Agentscited by 248
  9. 2024FrontierMath: A Benchmark for Evaluating Advanced Mathematical Reasoning in AIcited by 243
  10. 2024Cybench: A Framework for Evaluating Cybersecurity Capabilities and Risks of Language Modelscited by 164
  11. 2025Measuring AI Ability to Complete Long Software Taskscited by 156
  12. 2024BALROG: Benchmarking Agentic LLM and VLM Reasoning On Gamescited by 146
  13. 2024Observational Scaling Laws and the Predictability of Language Model Performancecited by 130
  14. 2022CommonsenseQA 2.0: Exposing the Limits of AI through Gamificationcited by 109
  15. 2025Lost in Benchmarks? Rethinking Large Language Model Benchmarking with Item Response Theorycited by 45
  16. 2024Why Has Predicting Downstream Capabilities of Frontier AI Models with Scale Remained Elusive?cited by 40
  17. 2024How predictable is language model benchmark performance?cited by 37
  18. 2024Training on the Test Task Confounds Evaluation and Emergencecited by 34
  19. 2024Algorithmic progress in language modelscited by 31
  20. 2025Reliable and Efficient Amortized Model-based Evaluationcited by 31
  21. 2025Adaptive Testing for LLM Evaluation: A Psychometric Alternative to Static Benchmarkscited by 28
  22. 2025GSO: Challenging Software Optimization Tasks for Evaluating SWE-Agentscited by 25
  23. 2025HCAST: Human-Calibrated Autonomy Software Taskscited by 24
  24. 2022Algorithmic progress in computer visioncited by 14
  25. 2025OSUniverse: Benchmark for Multimodal GUI-navigation AI Agentscited by 10
  26. 2025Factorio Learning Environmentcited by 7
  27. 2018Analysing Results from AI Benchmarks: Key Indicators and How to Obtain Themcited by 6
  28. 2025Forecasting AI Time Horizon Under Compute Slowdownscited by 3

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 36.

36 papers.

Sort:

By area:

  1. 2026Beyond Benchmarks: MathArena as an Evaluation Platform for Mathematics with LLMscited by 137
  2. 2026Think Fast: Estimating No-CoT Task-Completion Time Horizons of Frontier AI Modelscited by 12
  3. 2026Agent psychometrics: Task-level performance prediction in agentic coding benchmarkscited by 11
  4. 2026Measuring AI R&D Automationcited by 9
  5. 2026Propensity Inference: Environmental Contributors to LLM Behaviourcited by 7
  6. 2026BRIDGE: Predicting Human Task Completion Time From Model Performancecited by 6
  7. 2026Capabilities Ain't All You Need: Measuring Propensities in AIcited by 6
  8. 2026Training Agents to Self-Report Misbehaviorcited by 6
  9. 2026From AGI to ASIcited by 4
  10. 2026ForecastBench-Sim: A Simulated-World Forecasting Benchmarkcited by 2
  11. 2026CurveShift: Is Agent Progress Scalar? Separating Level from Shapecited by 2
  12. 2026Are AI Capabilities Increasing Exponentially? A Competing Hypothesiscited by 1
24 more
  1. 2026AI Revealed Preferencescited by 1
  2. 2026SecProbe: Adaptive Evaluation of Coding Agents on Cybersecurity Vulnerabilitiescited by 1
  3. 2026What if automating AI R&D triggers an intelligence explosion?cited by 1
  4. 2026Dynamic LLM Routers are Often Misguidedcited by 1
  5. 2026Punctuated Equilibria in Artificial Intelligence: The Institutional Scaling Law and the Speciation of Sovereign AIcited by 0
  6. 2026A Comparative Study in Surgical AI: Potential and Limitations of Data, Compute, and Scalingcited by 0
  7. 2026Growing Pains: Extensible and Efficient LLM Benchmarking Via Fixed Parameter Calibrationcited by 0
  8. 2026Agent Island: A Saturation- and Contamination-Resistant Benchmark from Multiagent Gamescited by 0
  9. 2026Is Capability a Liability? More Capable Language Models Make Worse Forecasts When It Matters Mostcited by 0
  10. 2026HakushoBench: A Japanese Chart and Table VQA Benchmark from Governmental White Paperscited by 0
  11. 2026Grounded Scaling: Why Agentic AI Needs Deterministic Environmentscited by 0
  12. 2026Two AI Metrics Diverged: Will it Make All the Difference?cited by 0
  13. 2026Harmonizing AI Safety Thresholdscited by 0
  14. 2026Economic Evaluations of Language Modelscited by 0
  15. 2026Messier: A High-Resolution Corpus for Cross-Benchmark Agent Evaluationcited by 0
  16. 2026The Closing Window: How Governments Could Lose Their Ability to Restrain Advanced AIcited by 0
  17. 2026Predicting Task Difficulty Without Rolloutscited by 0
  18. 2026Frontier AI Forecasting Has a Measurement Problem: An Audit of Progress Evidencecited by 0
  19. 2026R3-Bench: LLMs Struggle with Resource-Rational Reasoning under Shared Budgetscited by 0
  20. 2026One Capability or Many? Structural and Predictive Tests of Benchmark Validity Disagree About Economic Benchmarks for Frontier AIcited by 0
  21. 2026The Price of Intelligence: A Quality-Adjusted Price Index for AI Servicescited by 0
  22. 2026Recursive Criticality of AI Self-Improvementcited by 0
  23. 2026Monitor Jailbreaking: Evading Chain-of-Thought Monitoring Without Encoded Reasoningcited by 0
  24. 2026Learning Steganography Is Easy, Learning Steganographic Reasoning Is Hardcited by 0