# Beyond One-Size-Fits-All: Tailored Benchmarks for Efficient Evaluation

arXiv: https://arxiv.org/abs/2502.13576 · HTML: https://mlsanity.com/paper/2502.13576/

## Key facts

- Cited by 12 papers in the map (data through 14 Oct 2026).
- Builds on 16 of its 28 references that are in the map.
- First posted on arXiv 19 Feb 2025.

## Builds on (most cited first, 16 of 16)

- [Training language models to follow instructions with human feedback](https://mlsanity.com/paper/2203.02155/) (2022)
- [Llama 2: Open Foundation and Fine-Tuned Chat Models](https://mlsanity.com/paper/2307.09288/) (2023)
- [Training Verifiers to Solve Math Word Problems](https://mlsanity.com/paper/2110.14168/) (2021)
- [Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challenge](https://mlsanity.com/paper/1803.05457/) (2018)
- [HellaSwag: Can a Machine Really Finish Your Sentence?](https://mlsanity.com/paper/1905.07830/) (2019)
- [Measuring Robustness to Natural Distribution Shifts in Image Classification](https://mlsanity.com/paper/2007.00644/) (2020)
- [Accuracy on the Line: On the Strong Correlation Between Out-of-Distribution and In-Distribution Generalization](https://mlsanity.com/paper/2107.04649/) (2021)
- [Observational Scaling Laws and the Predictability of Language Model Performance](https://mlsanity.com/paper/2405.10938/) (2024)
- [Agreement-on-the-Line: Predicting the Performance of Neural Networks under Distribution Shift](https://mlsanity.com/paper/2206.13089/) (2022)
- [Anchor Points: Benchmarking Models with Much Fewer Examples](https://mlsanity.com/paper/2309.08638/) (2023)
- [Efficient Benchmarking of Language Models](https://mlsanity.com/paper/2308.11696/) (2023)
- [Predicting Emergent Abilities with Infinite Resolution Evaluation](https://mlsanity.com/paper/2310.03262/) (2023)
- [100 instances is all you need: predicting the success of a new LLM on unseen data by testing on a few instances](https://mlsanity.com/paper/2409.03563/) (2024)
- [Exploring The Landscape of Distributional Robustness for Question Answering Models](https://mlsanity.com/paper/2210.12517/) (2022)
- [Data Efficient Evaluation of Large Language Models and Text-to-Image Models via Adaptive Sampling](https://mlsanity.com/paper/2406.15527/) (2024)
- [Predicting the Performance of Foundation Models via Agreement-on-the-Line](https://mlsanity.com/paper/2404.01542/) (2024)

## Built on by (most cited first, 12 of 12)

- [SparseEval: Efficient Evaluation of Large Language Models by Sparse Optimization](https://mlsanity.com/paper/2602.07909/) (2026)
- [PairBench: Are Vision-Language Models Reliable at Comparing What They See?](https://mlsanity.com/paper/2502.15210/) (2025)
- [How Foundational are Foundation Models for Time Series Forecasting?](https://mlsanity.com/paper/2510.00742/) (2025)
- [DISCO: Diversifying Sample Condensation for Efficient Model Evaluation](https://mlsanity.com/paper/2510.07959/) (2025)
- [LoopArena: Benchmarking Models as Runtime Controllers for Loop Engineering](https://mlsanity.com/paper/2608.28281/) (2026)
- [Learning More from Less: Unlocking Internal Representations for Benchmark Compression](https://mlsanity.com/paper/2602.00710/) (2026)
- [Efficient Benchmarking Is Just Feature Selection and Multiple Regression](https://mlsanity.com/paper/2605.25773/) (2026)
- [Submodular Evaluation Subset Selection in Automatic Prompt Optimization](https://mlsanity.com/paper/2601.03493/) (2026)
- [Efficient SWE Agent Benchmarking via Trajectory-Aware Evaluation](https://mlsanity.com/paper/2609.01603/) (2026)
- [Zipbench: Low-Cost Framework for Compressing Comprehensive Benchmarks of Large Language Models](https://mlsanity.com/paper/2609.12475/) (2026)
- [Beyond Outcomes: Dual-View Relational Learning for Efficient Agent Benchmarking](https://mlsanity.com/paper/2609.18909/) (2026)
- [Are We Testing Too Much? A Survey of Benchmark Compression for Efficient Evaluation of Large Language Models](https://mlsanity.com/paper/2610.18248/) (2026)

## Abstract (arXiv)

Evaluating models on large benchmarks is very resource-intensive, especially during the period of rapid model evolution. Existing efficient evaluation methods estimate the performance of target models by testing them only on a small and static coreset of the benchmark, which is derived from the publicly available evaluation results of source models. These methods rely on the assumption that target models have high prediction consistency with source models. However, we demonstrate that it doesn't generalize well in practice. To alleviate the inconsistency issue, we present TailoredBench, a method that conducts customized evaluation tailored to each target model. Specifically, a Global-coreset is first constructed as a probe to identify the most consistent source models for each target model with an adaptive source model selection strategy. Afterwards, a scalable K-Medoids clustering algorithm is proposed to extend the Global-coreset to a tailored Native-coreset for each target model. According to the predictions on Native-coresets, we obtain the performance of target models on the whole benchmark with a calibrated estimation strategy. Comprehensive experiments on 5 benchmarks across over 300 models demonstrate that compared to best performing baselines, TailoredBench achieves an average reduction of 31.4% in MAE of accuracy estimates under the same inference budgets, showcasing strong effectiveness and generalizability.