# SparseEval: Efficient Evaluation of Large Language Models by Sparse Optimization

arXiv: https://arxiv.org/abs/2602.07909 · HTML: https://mlsanity.com/paper/2602.07909/

## Key facts

- Cited by 8 papers in the map (data through 14 Oct 2026).
- Builds on 18 of its 22 references that are in the map.
- First posted on arXiv 8 Feb 2026.

## Builds on (most cited first, 18 of 18)

- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [GPT-4 Technical Report](https://mlsanity.com/paper/2303.08774/) (2023)
- [Training Verifiers to Solve Math Word Problems](https://mlsanity.com/paper/2110.14168/) (2021)
- [Measuring Massive Multitask Language Understanding](https://mlsanity.com/paper/2009.03300/) (2020)
- [Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challenge](https://mlsanity.com/paper/1803.05457/) (2018)
- [HellaSwag: Can a Machine Really Finish Your Sentence?](https://mlsanity.com/paper/1905.07830/) (2019)
- [Qwen Technical Report](https://mlsanity.com/paper/2309.16609/) (2023)
- [TruthfulQA: Measuring How Models Mimic Human Falsehoods](https://mlsanity.com/paper/2109.07958/) (2021)
- [UMAP: Uniform Manifold Approximation and Projection for Dimension Reduction](https://mlsanity.com/paper/1802.03426/) (2018)
- [Qwen2 Technical Report](https://mlsanity.com/paper/2407.10671/) (2024)
- [LIMO: Less is More for Reasoning](https://mlsanity.com/paper/2502.03387/) (2025)
- [tinyBenchmarks: evaluating LLMs with fewer examples](https://mlsanity.com/paper/2402.14992/) (2024)
- [LIMR: Less is More for RL Scaling](https://mlsanity.com/paper/2502.11886/) (2025)
- [Anchor Points: Benchmarking Models with Much Fewer Examples](https://mlsanity.com/paper/2309.08638/) (2023)
- [Efficient Benchmarking of Language Models](https://mlsanity.com/paper/2308.11696/) (2023)
- [100 instances is all you need: predicting the success of a new LLM on unseen data by testing on a few instances](https://mlsanity.com/paper/2409.03563/) (2024)
- [Beyond One-Size-Fits-All: Tailored Benchmarks for Efficient Evaluation](https://mlsanity.com/paper/2502.13576/) (2025)
- [EffiEval: Efficient and Generalizable Model Evaluation via Capability Coverage Maximization](https://mlsanity.com/paper/2508.09662/) (2025)

## Built on by (most cited first, 8 of 8)

- [PACE: A Proxy for Agentic Capability Evaluation](https://mlsanity.com/paper/2607.02032/) (2026)
- [MINCE: Shrinking LLM Evaluation Datasets via Few-Model Monte Carlo Calibration](https://mlsanity.com/paper/2606.22826/) (2026)
- [Zipbench: Low-Cost Framework for Compressing Comprehensive Benchmarks of Large Language Models](https://mlsanity.com/paper/2609.12475/) (2026)
- [Beyond Outcomes: Dual-View Relational Learning for Efficient Agent Benchmarking](https://mlsanity.com/paper/2609.18909/) (2026)
- [Hierarchical Compression of Vision-Language Model Benchmarks](https://mlsanity.com/paper/2609.37515/) (2026)
- [cua-speedrun: Standardized Benchmarking of the Speed of Computer-Use Agents](https://mlsanity.com/paper/2609.40284/) (2026)
- [Are We Testing Too Much? A Survey of Benchmark Compression for Efficient Evaluation of Large Language Models](https://mlsanity.com/paper/2610.18248/) (2026)
- [EvalBuilder: Can LLMs Build Efficient Benchmarks?](https://mlsanity.com/paper/2610.20099/) (2026)

## Abstract (arXiv)

As large language models (LLMs) continue to scale up, their performance on various downstream tasks has significantly improved. However, evaluating their capabilities has become increasingly expensive, as performing inference on a large number of benchmark samples incurs high computational costs. In this paper, we revisit the model-item performance matrix and show that it exhibits sparsity, that representative items can be selected as anchors, and that the task of efficient benchmarking can be formulated as a sparse optimization problem. Based on these insights, we propose SparseEval, a method that, for the first time, adopts gradient descent to optimize anchor weights and employs an iterative refinement strategy for anchor selection. We utilize the representation capacity of MLP to handle sparse optimization and propose the Anchor Importance Score and Candidate Importance Score to evaluate the value of each item for task-aware refinement. Extensive experiments demonstrate the low estimation error and high Kendall's~τ of our method across a variety of benchmarks, showcasing its superior robustness and practicality in real-world scenarios. Code is available at https://github.com/taolinzhang/SparseEval.