mlsanityData to 14 Oct 2026

You Don't Need to Run Every Eval

arXiv 2606.24020 ↗First posted 22 Jun 2026cs.LGcited by 3

Not on the map. Why?

Abstract (arXiv)

A modern model release reports scores on 40+ benchmarks and the same evaluations were run many more times before it: to track training progress, compare design choices, and select the checkpoint for the release. But do we need to run every eval? We compile a public score matrix of 84 frontier models on 133 benchmarks (2,604 cells, 23.3% filled) and find it is approximately rank-2: a model's scores across all benchmarks are largely determined by just two numbers. We confirm this in two ways: (i) scores hidden from the matrix are best recovered using two factors, and (ii) two factors already explain over 90% of the variation among models on the benchmarks they share. Building on this, we design BenchPress: a logit-space rank-2 matrix completion method that recovers held-out scores to within 4.6 points. Using BenchPress, we find a subset of five benchmarks {GPQA-D, HLE, Codeforces, MMLU-Pro, ARC-AGI-1} that can recover the rest of a model's public scorecard to within 3.93 points. For a tighter evaluation budget, a cheaper set {GPQA-D, MMLU-Pro, Aider Polyglot, MATH-500, AIME 2026} can predict a model's evals to within 4.55. Finally, we identify what affects prediction reliability and combine these factors with predictor disagreement to quantify when predictions can be trusted. We release the score matrix, the code for reproducing all of our experiments on Github.

Key facts. Cited by 3 papers in the map (data through 14 Oct 2026). Builds on 29 of its 49 references that are in the map. First posted on arXiv 22 Jun 2026.

Builds on

29 of 49 references.

References · 29

  1. 2020Measuring Massive Multitask Language Understandingcited by 7,963
  2. 2023Let's Verify Step by Stepcited by 3,789
  3. 2023SWE-bench: Can Language Models Resolve Real-World GitHub Issues?cited by 3,373
  4. 2023GPQA: A Graduate-Level Google-Proof Q&A Benchmarkcited by 2,879
  5. 2024Chatbot Arena: An Open Platform for Evaluating LLMs by Human Preferencecited by 1,184
  6. 2026DeepSeek-V4: Towards Highly Efficient Million-Token Context Intelligencecited by 1,040
  7. 2024tinyBenchmarks: evaluating LLMs with fewer examplescited by 310
  8. 2025MathArena: Evaluating LLMs on Uncontaminated Math Competitionscited by 297
  9. 2024LiveBench: A Challenging, Contamination-Limited LLM Benchmarkcited by 262
  10. 2024Lessons from the Trenches on Reproducible Evaluation of Language Modelscited by 179
  11. 2025GDPval: Evaluating AI Model Performance on Real-World Economically Valuable Taskscited by 158
  12. 2024Observational Scaling Laws and the Predictability of Language Model Performancecited by 130
17 more
  1. 2023Anchor Points: Benchmarking Models with Much Fewer Examplescited by 86
  2. 2024MixEval: Deriving Wisdom of the Crowd from LLM Benchmark Mixturescited by 74
  3. 2023Efficient Benchmarking of Language Modelscited by 71
  4. 2023Revealing the structure of language model capabilitiescited by 40
  5. 2024metabench -- A Sparse Benchmark of Reasoning and Knowledge in Large Language Modelscited by 39
  6. 2023How Predictable Are Large Language Model Capabilities? A Case Study on BIG-benchcited by 34
  7. 2025How Benchmark Prediction from Fewer Data Misses the Markcited by 28
  8. 2023Evidence of interrelated cognitive-like capabilities in large language models: Indications of artificial general intelligence or achievement?cited by 27
  9. 2024Sloth: scaling laws for LLM skills to predict multi-benchmark performance across familiescited by 24
  10. 2024On Speeding Up Language Model Evaluationcited by 15
  11. 2025Train-before-Test Harmonizes Language Model Rankingscited by 12
  12. 2025Scales++: Compute Efficient Evaluation Subset Selection with Cognitive Scales Embeddingscited by 11
  13. 2024Collaborative Performance Prediction for Large Language Modelscited by 9
  14. 2025Rethinking LLM Evaluation: Can We Evaluate LLMs with 200x Less Data?cited by 8
  15. 2025Predicting LLM Reasoning Performance with Small Proxy Modelcited by 5
  16. 2025DISCO: Diversifying Sample Condensation for Efficient Model Evaluationcited by 5
  17. 2026Every Eval Ever: A Unifying Schema and Community Repository for AI Evaluation Resultscited by 3

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 3.

3 papers.