# SimBA: Simplifying Benchmark Analysis Using Performance Matrices Alone

arXiv: https://arxiv.org/abs/2510.17998 · HTML: https://mlsanity.com/paper/2510.17998/

## Key facts

- Cited by 6 papers in the map (data through 14 Oct 2026).
- Builds on 15 of its 33 references that are in the map.
- First posted on arXiv 20 Oct 2025.

## Builds on (most cited first, 15 of 15)

- [Measuring Massive Multitask Language Understanding](https://mlsanity.com/paper/2009.03300/) (2020)
- [Understanding Black-box Predictions via Influence Functions](https://mlsanity.com/paper/1703.04730/) (2017)
- [Deep Learning on a Data Diet: Finding Important Examples Early in Training](https://mlsanity.com/paper/2107.07075/) (2021)
- [tinyBenchmarks: evaluating LLMs with fewer examples](https://mlsanity.com/paper/2402.14992/) (2024)
- [GRAD-MATCH: Gradient Matching based Data Subset Selection for Efficient Deep Model Training](https://mlsanity.com/paper/2103.00123/) (2021)
- [Scaling Up Influence Functions](https://mlsanity.com/paper/2112.03052/) (2021)
- [Anchor Points: Benchmarking Models with Much Fewer Examples](https://mlsanity.com/paper/2309.08638/) (2023)
- [Efficient Benchmarking of Language Models](https://mlsanity.com/paper/2308.11696/) (2023)
- [Evaluating Evaluation Metrics: A Framework for Analyzing NLG Evaluation Metrics using Measurement Theory](https://mlsanity.com/paper/2305.14889/) (2023)
- [DataDecide: How to Predict Best Pretraining Data with Small Experiments](https://mlsanity.com/paper/2504.11393/) (2025)
- [How Predictable Are Large Language Model Capabilities? A Case Study on BIG-bench](https://mlsanity.com/paper/2305.14947/) (2023)
- [A Coreset Selection of Coreset Selection Literature: Introduction and Recent Advances](https://mlsanity.com/paper/2505.17799/) (2025)
- [How to Select Datapoints for Efficient Human Evaluation of NLG Models?](https://mlsanity.com/paper/2501.18251/) (2025)
- [MICE for CATs: Model-Internal Confidence Estimation for Calibrating Agents with Tools](https://mlsanity.com/paper/2504.20168/) (2025)
- [Chasing Random: Instruction Selection Strategies Fail to Generalize](https://mlsanity.com/paper/2410.15225/) (2024)

## Built on by (most cited first, 6 of 6)

- [Efficient Benchmarking of AI Agents](https://mlsanity.com/paper/2603.23749/) (2026)
- [Offline Preference-Based Trajectory Evaluation](https://mlsanity.com/paper/2606.17541/) (2026)
- [Coresets Before Score Sets: Evaluation-Unsupervised Prompt Subset Selection for LLM Benchmarks](https://mlsanity.com/paper/2607.09739/) (2026)
- [The ACUTE Protocol: Operationalizing Language Model Activations for Better Calibration, Utility, and Trust](https://mlsanity.com/paper/2606.07822/) (2026)
- [Harnessing the Latent Space: From Steering Vectors to Model Calibrators for Control and Trust](https://mlsanity.com/paper/2607.00083/) (2026)
- [Are We Testing Too Much? A Survey of Benchmark Compression for Efficient Evaluation of Large Language Models](https://mlsanity.com/paper/2610.18248/) (2026)

## Abstract (arXiv)

Modern language models are evaluated on large benchmarks, which are difficult to make sense of, especially for model selection. Looking at the raw evaluation numbers themselves using a model-centric lens, we propose SimBA, a three phase framework to Simplify Benchmark Analysis. The three phases of SimBA are: stalk, where we conduct dataset & model comparisons, prowl, where we discover a representative subset, and pounce, where we use the representative subset to predict performance on a held-out set of models. Applying SimBA to three popular LM benchmarks: HELM, MMLU, and BigBenchLite reveals that across all three benchmarks, datasets and models relate strongly to one another (stalk). We develop an representative set discovery algorithm which covers a benchmark using raw evaluation scores alone. Using our algorithm, we find that with 6.25% (1/16), 1.7% (1/58), and 28.4% (21/74) of the datasets for HELM, MMLU, and BigBenchLite respectively, we achieve coverage levels of at least 95% (prowl). Additionally, using just these representative subsets, we can both preserve model ranks and predict performance on a held-out set of models with near zero mean-squared error (pounce). Taken together, SimBA can help model developers improve efficiency during model training and dataset creators validate whether their newly created dataset differs from existing datasets in a benchmark. Our code is open source, available at https://github.com/nishantsubramani/simba.