# BenTo: Benchmark Task Reduction with In-Context Transferability

arXiv: https://arxiv.org/abs/2410.13804 · HTML: https://mlsanity.com/paper/2410.13804/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- Builds on 27 of its 33 references that are in the map.
- First posted on arXiv 17 Oct 2024.

## Builds on (most cited first, 27 of 27)

- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [Llama 2: Open Foundation and Fine-Tuned Chat Models](https://mlsanity.com/paper/2307.09288/) (2023)
- [Measuring Massive Multitask Language Understanding](https://mlsanity.com/paper/2009.03300/) (2020)
- [Finetuned Language Models Are Zero-Shot Learners](https://mlsanity.com/paper/2109.01652/) (2021)
- [Beyond the Imitation Game: Quantifying and extrapolating the capabilities of language models](https://mlsanity.com/paper/2206.04615/) (2022)
- [Phi-3 Technical Report: A Highly Capable Language Model Locally on Your Phone](https://mlsanity.com/paper/2404.14219/) (2024)
- [Challenging BIG-Bench Tasks and Whether Chain-of-Thought Can Solve Them](https://mlsanity.com/paper/2210.09261/) (2022)
- [A Survey on Evaluation of Large Language Models](https://mlsanity.com/paper/2307.03109/) (2023)
- [Gemma: Open Models Based on Gemini Research and Technology](https://mlsanity.com/paper/2403.08295/) (2024)
- [Aligning AI With Shared Human Values](https://mlsanity.com/paper/2008.02275/) (2020)
- [AGIEval: A Human-Centric Benchmark for Evaluating Foundation Models](https://mlsanity.com/paper/2304.06364/) (2023)
- [TinyLlama: An Open-Source Small Language Model](https://mlsanity.com/paper/2401.02385/) (2024)
- [LESS: Selecting Influential Data for Targeted Instruction Tuning](https://mlsanity.com/paper/2402.04333/) (2024)
- [tinyBenchmarks: evaluating LLMs with fewer examples](https://mlsanity.com/paper/2402.14992/) (2024)
- [From Quantity to Quality: Boosting LLM Performance with Self-Guided Data Selection for Instruction Tuning](https://mlsanity.com/paper/2308.12032/) (2023)
- [LEEP: A New Measure to Evaluate Transferability of Learned Representations](https://mlsanity.com/paper/2002.12462/) (2020)
- [Superfiltering: Weak-to-Strong Data Filtering for Fast Instruction-Tuning](https://mlsanity.com/paper/2402.00530/) (2024)
- [Exploring and Predicting Transferability across NLP Tasks](https://mlsanity.com/paper/2005.00770/) (2020)
- [Stable LM 2 1.6B Technical Report](https://mlsanity.com/paper/2402.17834/) (2024)
- [Anchor Points: Benchmarking Models with Much Fewer Examples](https://mlsanity.com/paper/2309.08638/) (2023)
- [An Information-Theoretic Approach to Transferability in Task Transfer Learning](https://mlsanity.com/paper/2212.10082/) (2022)
- [Efficient Benchmarking of Language Models](https://mlsanity.com/paper/2308.11696/) (2023)
- [Inadequacies of Large Language Model Benchmarks in the Era of Generative Artificial Intelligence](https://mlsanity.com/paper/2402.09880/) (2024)
- [OTCE: A Transferability Metric for Cross-Domain Cross-Task Representations](https://mlsanity.com/paper/2103.13843/) (2021)
- [Transferability in Deep Learning: A Survey](https://mlsanity.com/paper/2201.05867/) (2022)
- [How Predictable Are Large Language Model Capabilities? A Case Study on BIG-bench](https://mlsanity.com/paper/2305.14947/) (2023)
- [OpenNMT: Neural Machine Translation Toolkit](https://mlsanity.com/paper/1805.11462/) (2018)

## Built on by (most cited first, 4 of 4)

- [How Reliable is Language Model Micro-Benchmarking?](https://mlsanity.com/paper/2510.08730/) (2025)
- [DISCO: Diversifying Sample Condensation for Efficient Model Evaluation](https://mlsanity.com/paper/2510.07959/) (2025)
- [Efficient Benchmarking Is Just Feature Selection and Multiple Regression](https://mlsanity.com/paper/2605.25773/) (2026)
- [Trajectory-Aware Benchmark Subset Selection for Cost-Efficient Software Engineering Agent Regression Testing](https://mlsanity.com/paper/2609.24928/) (2026)

## Abstract (arXiv)

Evaluating large language models (LLMs) is costly: it requires the generation and examination of LLM outputs on a large-scale benchmark of various tasks. This paper investigates how to efficiently reduce the tasks used to benchmark LLMs without affecting the evaluation quality. Our study reveals that task transferability and relevance provide critical information to identify the most representative subset of tasks via optimizing a facility location function. We propose a practically efficient metric for estimating the transferability between two tasks via in-context learning (ICL). By analyzing the pairwise transferability, we can reduce tasks in a modern LLM benchmark (e.g., MMLU or FLAN) to 5% while inducing only a <4% difference to the evaluation on the original benchmark. Compared to prior works, our method is training-free, gradient-free, and highly efficient requiring ICL only.