mlsanityData to 7 Oct 2026

metabench -- A Sparse Benchmark of Reasoning and Knowledge in Large Language Models

arXiv 2407.12844 ↗First posted 4 Jul 2024cs.CL cs.LG stat.MLcited by 35

On the map: LLM reasoning → Reasoning benchmarks and evaluation

Abstract (arXiv)

Large Language Models (LLMs) vary in their abilities on a range of tasks. Initiatives such as the Open LLM Leaderboard aim to quantify these differences with several large benchmarks (sets of test items to which an LLM can respond either correctly or incorrectly). However, high correlations within and between benchmark scores suggest that (1) there exists a small set of common underlying abilities that these benchmarks measure, and (2) items tap into redundant information and the benchmarks may thus be considerably compressed. We use data from n > 5000 LLMs to identify the most informative items of six benchmarks, ARC, GSM8K, HellaSwag, MMLU, TruthfulQA and WinoGrande (with d = 28,632 items in total). From them we distill a sparse benchmark, metabench, that has less than 3% of the original size of all six benchmarks combined. This new sparse benchmark goes beyond point scores by yielding estimators of the underlying benchmark-specific abilities. We show that these estimators (1) can be used to reconstruct each original individual benchmark score with, on average, 1.24% root mean square error (RMSE), (2) reconstruct the original total score with 0.58% RMSE, and (3) have a single underlying common factor whose Spearman correlation with the total score is r = 0.94.

Key facts. Cited by 35 papers in the map (data through 7 Oct 2026). Placed in Reasoning benchmarks and evaluation, one of 17 directions in LLM reasoning. Builds on 11 of its 93 references that are in the map. First posted on arXiv 4 Jul 2024.

Builds on

11 of 93 references.

References · 11

  1. 2021Training Verifiers to Solve Math Word Problemscited by 9,743
  2. 2021Evaluating Large Language Models Trained on Codecited by 8,053
  3. 2021On the Opportunities and Risks of Foundation Modelscited by 3,961
  4. 2023Sparks of Artificial General Intelligence: Early experiments with GPT-4cited by 2,579
  5. 2022Holistic Evaluation of Language Modelscited by 115
  6. 2023CritiqueLLM: Towards an Informative Critique Generation Model for Evaluation of Large Language Model Generationcited by 65
  7. 2024CogBench: a large language model walks into a psychology labcited by 60
  8. 2023Automating Customer Service using LangChain: Building custom open-source GPT Chatbot for organizationscited by 59
  9. 2023Position: AI Evaluation Should Learn from How We Test Humanscited by 36
  10. 2023Who is ChatGPT? Benchmarking LLMs' Psychological Portrayal Using PsychoBenchcited by 28
  11. 2023Inferring Capabilities from Task Performance with Bayesian Triangulationcited by 13

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 10; 2026: 25.

35 papers.

Sort:

By area:

  1. 2025Measuring AI Ability to Complete Long Software Taskscited by 142
  2. 2025Fluid Language Model Benchmarkingcited by 30
  3. 2025How Benchmark Prediction from Fewer Data Misses the Markcited by 26
  4. 2025Adaptive Testing for LLM Evaluation: A Psychometric Alternative to Static Benchmarkscited by 25
  5. 2025Neural Architecture Discovery via Autonomous Evolutioncited by 15
  6. 2025Scales++: Compute Efficient Evaluation Subset Selection with Cognitive Scales Embeddingscited by 9
  7. 2025Rethinking LLM Evaluation: Can We Evaluate LLMs with 200x Less Data?cited by 8
  8. 2025Latency-Response Theory Model: Evaluating Large Language Models via Response Accuracy and Chain-of-Thought Lengthcited by 5
  9. 2025DISCO: Diversifying Sample Condensation for Efficient Model Evaluationcited by 4
  10. 2026Confident Rankings with Fewer Items: Adaptive LLM Evaluation with Continuous Scorescited by 4
  11. 2026Can We Trust Item Response Theory for AI Evaluation?cited by 3
  12. 2026Measuring What AI Systems Might Do: Towards A Measurement Science in AIcited by 2
23 more
  1. 2026Evaluating Cross-Modal Reasoning Ability and Problem Characteristics with Multimodal Item Response Theorycited by 2
  2. 2026ProEval: Proactive Failure Discovery and Efficient Performance Estimation for Generative AI Evaluationcited by 2
  3. 2026PACE: A Proxy for Agentic Capability Evaluationcited by 2
  4. 2025Can a Crow Hatch a Falcon? Lineage Matters in Predicting Large Language Model Performancecited by 1
  5. 2026Cost-Efficient Estimation of General Abilities Across Benchmarkscited by 1
  6. 2026Select Smarter, Not More: Prompt-Aware Evaluation Scheduling with Submodular Guaranteescited by 1
  7. 2026Efficient Benchmarking Is Just Feature Selection and Multiple Regressioncited by 1
  8. 2026Cutting LLM Evaluation Costs with SySRs: A Bandit Algorithm that Provably Exploits Model Similaritycited by 1
  9. 2026You Don't Need to Run Every Evalcited by 1
  10. 2026Stochastic Sampling is Epistemically Shallow: The Dimensionality Gap Between Temperature Variation and Model Diversity in LLMscited by 1
  11. 2026Harbor Adapters and Harbor-Index: Infrastructure and a Curated Meta-Dataset for Large-Scale Agentic Evaluationcited by 1
  12. 2026Aligning Language Model Benchmarks with Pairwise Preferencescited by 0
  13. 2026Capturing LLM Capabilities via Evidence-Calibrated Query Clusteringcited by 0
  14. 2026Item Response Scaling Laws: A Measurement Theory Approach for Efficient and Generalizable Neural Scaling Estimationcited by 0
  15. 2026FailureScope: Cross-Regime Behavioral Diagnosis of Language Model Weaknessescited by 0
  16. 2026AGC-Bench: Measuring Artificial General Creativitycited by 0
  17. 2026Beyond Independent Labels: Schwartz-Geometry Decoding for Human Value Detectioncited by 0
  18. 2026BayesAME: Bayesian Active Model Evaluationcited by 0
  19. 2026Efficient SWE Agent Benchmarking via Trajectory-Aware Evaluationcited by 0
  20. 2026Accounting for Bias Enables Sustainable LLM Evaluationcited by 0
  21. 2026Hierarchical Compression of Vision-Language Model Benchmarkscited by 0
  22. 2026cua-speedrun: Standardized Benchmarking of the Speed of Computer-Use Agentscited by 0
  23. 2026What Makes Something Hard(er)? Explaining Question Difficulty in Natural Languagecited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2023Chain-of-Thought Hub: A Continuous Effort to Measure Large Language Models' Reasoning Performance
  2. 2025ReasoningWeekly: A General Knowledge and Verbal Reasoning Challenge for Large Language Models
  3. 2025Beyond Chains of Thought: Benchmarking Latent-Space Reasoning Abilities in Large Language Models
  4. 2024Beyond Accuracy: Evaluating the Reasoning Behavior of Large Language Models -- A Survey
  5. 2025LingBench++: A Linguistically-Informed Benchmark and Reasoning Framework for Multi-Step and Cross-Cultural Inference with LLMs
  6. 2026Understanding Axes of Difficulty For Long Context Tasks Via PredicateLongBench
  7. 2025Towards Reasoning Ability of Small Language Models
  8. 2025Domain Specific Benchmarks for Evaluating Multimodal Large Language Models