# You Don't Need to Run Every Eval

arXiv: https://arxiv.org/abs/2606.24020 · HTML: https://mlsanity.com/paper/2606.24020/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Builds on 29 of its 49 references that are in the map.
- First posted on arXiv 22 Jun 2026.

## Builds on (most cited first, 29 of 29)

- [Measuring Massive Multitask Language Understanding](https://mlsanity.com/paper/2009.03300/) (2020)
- [Let's Verify Step by Step](https://mlsanity.com/paper/2305.20050/) (2023)
- [SWE-bench: Can Language Models Resolve Real-World GitHub Issues?](https://mlsanity.com/paper/2310.06770/) (2023)
- [GPQA: A Graduate-Level Google-Proof Q&A Benchmark](https://mlsanity.com/paper/2311.12022/) (2023)
- [Chatbot Arena: An Open Platform for Evaluating LLMs by Human Preference](https://mlsanity.com/paper/2403.04132/) (2024)
- [DeepSeek-V4: Towards Highly Efficient Million-Token Context Intelligence](https://mlsanity.com/paper/2606.19348/) (2026)
- [tinyBenchmarks: evaluating LLMs with fewer examples](https://mlsanity.com/paper/2402.14992/) (2024)
- [MathArena: Evaluating LLMs on Uncontaminated Math Competitions](https://mlsanity.com/paper/2505.23281/) (2025)
- [LiveBench: A Challenging, Contamination-Limited LLM Benchmark](https://mlsanity.com/paper/2406.19314/) (2024)
- [Lessons from the Trenches on Reproducible Evaluation of Language Models](https://mlsanity.com/paper/2405.14782/) (2024)
- [GDPval: Evaluating AI Model Performance on Real-World Economically Valuable Tasks](https://mlsanity.com/paper/2510.04374/) (2025)
- [Observational Scaling Laws and the Predictability of Language Model Performance](https://mlsanity.com/paper/2405.10938/) (2024)
- [Anchor Points: Benchmarking Models with Much Fewer Examples](https://mlsanity.com/paper/2309.08638/) (2023)
- [MixEval: Deriving Wisdom of the Crowd from LLM Benchmark Mixtures](https://mlsanity.com/paper/2406.06565/) (2024)
- [Efficient Benchmarking of Language Models](https://mlsanity.com/paper/2308.11696/) (2023)
- [Revealing the structure of language model capabilities](https://mlsanity.com/paper/2306.10062/) (2023)
- [metabench -- A Sparse Benchmark of Reasoning and Knowledge in Large Language Models](https://mlsanity.com/paper/2407.12844/) (2024)
- [How Predictable Are Large Language Model Capabilities? A Case Study on BIG-bench](https://mlsanity.com/paper/2305.14947/) (2023)
- [How Benchmark Prediction from Fewer Data Misses the Mark](https://mlsanity.com/paper/2506.07673/) (2025)
- [Evidence of interrelated cognitive-like capabilities in large language models: Indications of artificial general intelligence or achievement?](https://mlsanity.com/paper/2310.11616/) (2023)
- [Sloth: scaling laws for LLM skills to predict multi-benchmark performance across families](https://mlsanity.com/paper/2412.06540/) (2024)
- [On Speeding Up Language Model Evaluation](https://mlsanity.com/paper/2407.06172/) (2024)
- [Train-before-Test Harmonizes Language Model Rankings](https://mlsanity.com/paper/2507.05195/) (2025)
- [Scales++: Compute Efficient Evaluation Subset Selection with Cognitive Scales Embeddings](https://mlsanity.com/paper/2510.26384/) (2025)
- [Collaborative Performance Prediction for Large Language Models](https://mlsanity.com/paper/2407.01300/) (2024)
- [Rethinking LLM Evaluation: Can We Evaluate LLMs with 200x Less Data?](https://mlsanity.com/paper/2510.10457/) (2025)
- [Predicting LLM Reasoning Performance with Small Proxy Model](https://mlsanity.com/paper/2509.21013/) (2025)
- [DISCO: Diversifying Sample Condensation for Efficient Model Evaluation](https://mlsanity.com/paper/2510.07959/) (2025)
- [Every Eval Ever: A Unifying Schema and Community Repository for AI Evaluation Results](https://mlsanity.com/paper/2606.14516/) (2026)

## Built on by (most cited first, 3 of 3)

- [Harbor Adapters and Harbor-Index: Infrastructure and a Curated Meta-Dataset for Large-Scale Agentic Evaluation](https://mlsanity.com/paper/2609.04298/) (2026)
- [Are We Testing Too Much? A Survey of Benchmark Compression for Efficient Evaluation of Large Language Models](https://mlsanity.com/paper/2610.18248/) (2026)
- [EvalBuilder: Can LLMs Build Efficient Benchmarks?](https://mlsanity.com/paper/2610.20099/) (2026)

## Abstract (arXiv)

A modern model release reports scores on 40+ benchmarks and the same evaluations were run many more times before it: to track training progress, compare design choices, and select the checkpoint for the release. But do we need to run every eval? We compile a public score matrix of 84 frontier models on 133 benchmarks (2,604 cells, 23.3% filled) and find it is approximately rank-2: a model's scores across all benchmarks are largely determined by just two numbers. We confirm this in two ways: (i) scores hidden from the matrix are best recovered using two factors, and (ii) two factors already explain over 90% of the variation among models on the benchmarks they share. Building on this, we design BenchPress: a logit-space rank-2 matrix completion method that recovers held-out scores to within 4.6 points. Using BenchPress, we find a subset of five benchmarks {GPQA-D, HLE, Codeforces, MMLU-Pro, ARC-AGI-1} that can recover the rest of a model's public scorecard to within 3.93 points. For a tighter evaluation budget, a cheaper set {GPQA-D, MMLU-Pro, Aider Polyglot, MATH-500, AIME 2026} can predict a model's evals to within 4.55. Finally, we identify what affects prediction reliability and combine these factors with predictor disagreement to quantify when predictions can be trusted. We release the score matrix, the code for reproducing all of our experiments on Github.