# A Survey on Large Language Model Benchmarks

arXiv: https://arxiv.org/abs/2508.15361 · HTML: https://mlsanity.com/paper/2508.15361/

## Key facts

- Cited by 36 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 21 Aug 2025.

## Built on by (most cited first, 30 of 36)

- [A Survey of Scientific Large Language Models: From Data Foundations to Agent Frontiers](https://mlsanity.com/paper/2508.21148/) (2025)
- [Reasoning in a Combinatorial and Constrained World: Benchmarking LLMs on Natural-Language Combinatorial Optimization](https://mlsanity.com/paper/2602.02188/) (2026)
- [ProbeLLM: Automating Principled Diagnosis of LLM Failures](https://mlsanity.com/paper/2602.12966/) (2026)
- [Quantization Undoes Alignment: Bias Emergence in Compressed LLMs Across Models and Precision Levels](https://mlsanity.com/paper/2605.15208/) (2026)
- [Geometric Metrics and LLMs: What They Measure and When They Work](https://mlsanity.com/paper/2509.25359/) (2025)
- [LexGenius: An Expert-Level Benchmark for Large Language Models in Legal General Intelligence](https://mlsanity.com/paper/2512.04578/) (2025)
- [Evaluation Cards: An Interpretive Layer for AI Evaluation Reporting](https://mlsanity.com/paper/2606.09809/) (2026)
- [Evaluating LLMs When They Do Not Know the Answer: Statistical Evaluation of Mathematical Reasoning via Comparative Signals](https://mlsanity.com/paper/2602.03061/) (2026)
- [Steering to Say No: Configurable Refusal via Activation Steering in Vision Language Models](https://mlsanity.com/paper/2602.07013/) (2026)
- [Mapping Overlaps in Benchmarks through Perplexity in the Wild](https://mlsanity.com/paper/2509.23488/) (2025)
- [LPDS: Evaluating LLM Robustness Through Logic-Preserving Difficulty Scaling](https://mlsanity.com/paper/2605.15393/) (2026)
- [MaP: A Unified Framework for Reliable Evaluation of Pre-training Dynamics](https://mlsanity.com/paper/2510.09295/) (2025)
- [Too Open for Opinion? Embracing Open-Endedness in Large Language Models for Social Simulation](https://mlsanity.com/paper/2510.13884/) (2025)
- [Charting the European LLM Benchmarking Landscape: A New Taxonomy and a Set of Best Practices](https://mlsanity.com/paper/2510.24450/) (2025)
- [Making LLMs Reliable When It Matters Most: A Five-Layer Architecture for High-Stakes Decisions](https://mlsanity.com/paper/2511.07669/) (2025)
- [Plausibility as Failure: How LLMs and Humans Co-Construct Epistemic Error](https://mlsanity.com/paper/2512.16750/) (2025)
- [How Hard is it to Rig a Benchmark? A Social Choice Analysis of Leaderboard Robustness](https://mlsanity.com/paper/2605.23628/) (2026)
- [MAVEN: Improving Generalization in Agentic Tool Calling](https://mlsanity.com/paper/2605.30738/) (2026)
- [How to Ask the AI: A User Perspective Survey for Large Language Model Prompting](https://mlsanity.com/paper/2608.07494/) (2026)
- [On Generalization in Agentic Tool Calling: CoreThink Agentic Reasoner and MAVEN Dataset](https://mlsanity.com/paper/2510.22898/) (2025)
- [Large language models for automated PRISMA 2020 adherence checking](https://mlsanity.com/paper/2511.16707/) (2025)
- [AfriEconQA: A Benchmark for Quantitative and Temporal Reasoning over World Bank Economic Reports](https://mlsanity.com/paper/2601.15297/) (2026)
- [ReLE: A Scalable System and Structured Benchmark for Diagnosing Capability Anisotropy in Chinese LLMs](https://mlsanity.com/paper/2601.17399/) (2026)
- [DEP: A Decentralized Large Language Model Evaluation Protocol](https://mlsanity.com/paper/2603.01167/) (2026)
- [SommBench: Assessing Sommelier Expertise of Language Models](https://mlsanity.com/paper/2603.12117/) (2026)
- [Can Large Language Models Reason and Optimize Under Constraints?](https://mlsanity.com/paper/2603.23004/) (2026)
- [Progressing beyond Art Masterpieces or Touristic Clich'es: how to assess your LLMs for cultural alignment?](https://mlsanity.com/paper/2604.25654/) (2026)
- [BenCSSmark: Making the Social Sciences Count in LLM Research](https://mlsanity.com/paper/2605.04886/) (2026)
- [Coordinates of Capability: A Unified MTMM-Geometric Framework for LLM Evaluation](https://mlsanity.com/paper/2605.08522/) (2026)
- [Efficient Ensemble Selection from Binary and Pairwise Feedback](https://mlsanity.com/paper/2605.09588/) (2026)

## Abstract (arXiv)

In recent years, with the rapid development of the depth and breadth of large language models' capabilities, various corresponding evaluation benchmarks have been emerging in increasing numbers. As a quantitative assessment tool for model performance, benchmarks are not only a core means to measure model capabilities but also a key element in guiding the direction of model development and promoting technological innovation. We systematically review the current status and development of large language model benchmarks for the first time, categorizing 283 representative benchmarks into three categories: general capabilities, domain-specific, and target-specific. General capability benchmarks cover aspects such as core linguistics, knowledge, and reasoning; domain-specific benchmarks focus on fields like natural sciences, humanities and social sciences, and engineering technology; target-specific benchmarks pay attention to risks, reliability, agents, etc. We point out that current benchmarks have problems such as inflated scores caused by data contamination, unfair evaluation due to cultural and linguistic biases, and lack of evaluation on process credibility and dynamic environments, and provide a referable design paradigm for future benchmark innovation.