# Label-Efficient Model Selection for Text Generation

arXiv: https://arxiv.org/abs/2402.07891 · HTML: https://mlsanity.com/paper/2402.07891/

## Key facts

- Cited by 19 papers in the map (data through 14 Oct 2026).
- Builds on 20 of its 39 references that are in the map.
- First posted on arXiv 12 Feb 2024.

## Builds on (most cited first, 20 of 20)

- [Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks](https://mlsanity.com/paper/1908.10084/) (2019)
- [Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena](https://mlsanity.com/paper/2306.05685/) (2023)
- [Don't Give Me the Details, Just the Summary! Topic-Aware Convolutional Neural Networks for Extreme Summarization](https://mlsanity.com/paper/1808.08745/) (2018)
- [Abstractive Text Summarization Using Sequence-to-Sequence RNNs and Beyond](https://mlsanity.com/paper/1602.06023/) (2016)
- [The NarrativeQA Reading Comprehension Challenge](https://mlsanity.com/paper/1712.07040/) (2017)
- [QuAC : Question Answering in Context](https://mlsanity.com/paper/1808.07036/) (2018)
- [State of What Art? A Call for Multi-Prompt LLM Evaluation](https://mlsanity.com/paper/2401.00595/) (2023)
- [Evaluation of Text Generation: A Survey](https://mlsanity.com/paper/2006.14799/) (2020)
- [Anchor Points: Benchmarking Models with Much Fewer Examples](https://mlsanity.com/paper/2309.08638/) (2023)
- [ACUTE-EVAL: Improved Dialogue Evaluation with Optimized Questions and Multi-turn Comparisons](https://mlsanity.com/paper/1909.03087/) (2019)
- [Active Testing: Sample-Efficient Model Evaluation](https://mlsanity.com/paper/2103.05331/) (2021)
- [Comparing Test Sets with Item Response Theory](https://mlsanity.com/paper/2106.00840/) (2021)
- [Let's Agree to Agree: Neural Networks Share Classification Order on Real Datasets](https://mlsanity.com/paper/1905.10854/) (2019)
- [Corpus Wide Argument Mining -- a Working Solution](https://mlsanity.com/paper/1911.10763/) (2019)
- [Benchmarking Large Language Model Capabilities for Conditional Generation](https://mlsanity.com/paper/2306.16793/) (2023)
- [Beyond User Self-Reported Likert Scale Ratings: A Comparison Model for Automatic Dialog Evaluation](https://mlsanity.com/paper/2005.10716/) (2020)
- [Active Bayesian Assessment for Black-Box Classifiers](https://mlsanity.com/paper/2002.06532/) (2020)
- [Active Evaluation: Efficient NLG Evaluation with Few Pairwise Comparisons](https://mlsanity.com/paper/2203.06063/) (2022)
- [PreQuEL: Quality Estimation of Machine Translation Outputs in Advance](https://mlsanity.com/paper/2205.09178/) (2022)
- [ALT-MAS: A Data-Efficient Framework for Active Testing of Machine Learning Algorithms](https://mlsanity.com/paper/2104.04999/) (2021)

## Built on by (most cited first, 19 of 19)

- [Efficient multi-prompt evaluation of LLMs](https://mlsanity.com/paper/2405.17202/) (2024)
- [JuStRank: Benchmarking LLM Judges for System Ranking](https://mlsanity.com/paper/2412.09569/) (2024)
- [How to Select Datapoints for Efficient Human Evaluation of NLG Models?](https://mlsanity.com/paper/2501.18251/) (2025)
- [AcTracer: Active Testing of Large Language Model via Multi-Stage Sampling](https://mlsanity.com/paper/2408.03573/) (2024)
- [Scaling Up Active Testing to Large Language Models](https://mlsanity.com/paper/2508.09093/) (2025)
- [How Reliable is Language Model Micro-Benchmarking?](https://mlsanity.com/paper/2510.08730/) (2025)
- [Learning on Model Weights using Tree Experts](https://mlsanity.com/paper/2410.13569/) (2024)
- [Compare without Despair: Reliable Preference Evaluation with Generation Separability](https://mlsanity.com/paper/2407.01878/) (2024)
- [Generative Active Testing: Efficient LLM Evaluation via Proxy Task Adaptation](https://mlsanity.com/paper/2603.19264/) (2026)
- [ProEval: Proactive Failure Discovery and Efficient Performance Estimation for Generative AI Evaluation](https://mlsanity.com/paper/2604.23099/) (2026)
- [Offline Preference-Based Trajectory Evaluation](https://mlsanity.com/paper/2606.17541/) (2026)
- [Stop Guessing When to Stop Testing: Efficient Model Evaluation with Just Enough Data](https://mlsanity.com/paper/2607.08522/) (2026)
- [Discovering Hidden Gems in Model Repositories](https://mlsanity.com/paper/2601.22157/) (2026)
- [Large Language Model Selection with Limited Annotations](https://mlsanity.com/paper/2605.24981/) (2026)
- [Prediction-Powered Active Testing](https://mlsanity.com/paper/2607.08347/) (2026)
- [Large Language Model Selection with Limited Annotations](https://mlsanity.com/paper/2510.09418/) (2025)
- [Fault-Tolerant Evaluation for Sample-Efficient Model Performance Estimators](https://mlsanity.com/paper/2602.07226/) (2026)
- [Which LLM to pick? Online Active Model Selection for Large Language Models](https://mlsanity.com/paper/2610.01592/) (2026)
- [Are We Testing Too Much? A Survey of Benchmark Compression for Efficient Evaluation of Large Language Models](https://mlsanity.com/paper/2610.18248/) (2026)

## Abstract (arXiv)

Model selection for a given target task can be costly, as it may entail extensive annotation of the quality of outputs of different models. We introduce DiffUse, an efficient method to make an informed decision between candidate text generation models based on preference annotations. DiffUse reduces the required amount of annotations, thus saving valuable time and resources in performing evaluation. DiffUse intelligently selects instances by clustering embeddings that represent the semantic differences between model outputs. Thus, it is able to identify a subset of examples that are more informative for preference decisions. Our method is model-agnostic, and can be applied to any text generation model for selecting between models, prompts and configurations. Moreover, we propose a practical iterative approach for dynamically determining how many instances to annotate. In a series of experiments over hundreds of model pairs, we demonstrate that DiffUse can dramatically reduce the required number of annotations -- by up to 75% -- while maintaining high evaluation reliability.