mlsanityData to 7 Oct 2026

On Speeding Up Language Model Evaluation

arXiv 2407.06172 ↗First posted 8 Jul 2024cs.AI cs.CLcited by 14

Not on the map. Why?

Abstract (arXiv)

Developing prompt-based methods with Large Language Models (LLMs) requires making numerous decisions, which give rise to a combinatorial search problem over hyper-parameters. This exhaustive evaluation can be time-consuming and costly. In this paper, we propose an textitadaptive approach to explore this space. We are exploiting the fact that often only few samples are needed to identify clearly superior or inferior settings, and that many evaluation tests are highly correlated. We lean on multi-armed bandits to sequentially identify the next (method, validation sample)-pair to evaluate and utilize low-rank matrix factorization to fill in missing evaluations. We carefully assess the efficacy of our approach on several competitive benchmark problems and show that it can identify the top-performing method using only 5-15% of the typical resources -- resulting in 85-95% LLM cost savings. Our code is available at https://github.com/kilian-group/banditeval.

Key facts. Cited by 14 papers in the map (data through 7 Oct 2026). Builds on 40 of its 66 references that are in the map. First posted on arXiv 8 Jul 2024.

Builds on

40 of 66 references.

References · 40

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  2. 2020Language Models are Few-Shot Learnerscited by 32,199
  3. 2021Training Verifiers to Solve Math Word Problemscited by 9,743
  4. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,393
  5. 2020Measuring Massive Multitask Language Understandingcited by 7,768
  6. 2019BERTScore: Evaluating Text Generation with BERTcited by 5,462
  7. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,304
  8. 2021Measuring Mathematical Problem Solving With the MATH Datasetcited by 5,203
  9. 2018GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understandingcited by 4,854
  10. 2018Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challengecited by 4,710
  11. 2022Large Language Models are Zero-Shot Reasonerscited by 4,323
  12. 2016SQuAD: 100,000+ Questions for Machine Comprehension of Textcited by 4,115
28 more
  1. 2019HellaSwag: Can a Machine Really Finish Your Sentence?cited by 4,018
  2. 2018HotpotQA: A Dataset for Diverse, Explainable Multi-hop Question Answeringcited by 3,686
  3. 2017TriviaQA: A Large Scale Distantly Supervised Challenge Dataset for Reading Comprehensioncited by 2,895
  4. 2019PIQA: Reasoning about Physical Commonsense in Natural Languagecited by 2,818
  5. 2022Solving Quantitative Reasoning Problems with Language Modelscited by 1,359
  6. 2018Don't Give Me the Details, Just the Summary! Topic-Aware Convolutional Neural Networks for Extreme Summarizationcited by 1,308
  7. 2019PubMedQA: A Dataset for Biomedical Research Question Answeringcited by 1,271
  8. 2023A Survey on Evaluation of Large Language Modelscited by 1,227
  9. 2020BLEURT: Learning Robust Metrics for Text Generationcited by 1,196
  10. 2024Chatbot Arena: An Open Platform for Evaluating LLMs by Human Preferencecited by 1,151
  11. 2021Measuring Coding Challenge Competence With APPScited by 879
  12. 2023Large Language Models as Optimizerscited by 830
  13. 2016Abstractive Text Summarization Using Sequence-to-Sequence RNNs and Beyondcited by 809
  14. 2021BARTScore: Evaluating Generated Text as Text Generationcited by 719
  15. 2021A Dataset of Information-Seeking Questions and Answers Anchored in Research Paperscited by 475
  16. 2021BOLD: Dataset and Metrics for Measuring Biases in Open-Ended Language Generationcited by 410
  17. 2023How Far Can Camels Go? Exploring the State of Instruction Tuning on Open Resourcescited by 405
  18. 2018Newsroom: A Dataset of 1.3 Million Summaries with Diverse Extractive Strategiescited by 292
  19. 2014On the Complexity of Best Arm Identification in Multi-Armed Bandit Modelscited by 280
  20. 2016Optimal Best Arm Identification with Fixed Confidencecited by 233
  21. 2018Nonconvex Optimization Meets Low-Rank Matrix Factorization: An Overviewcited by 159
  22. 2013lil' UCB : An Optimal Exploration Algorithm for Multi-Armed Banditscited by 148
  23. 2024Mapping the Increasing Use of LLMs in Scientific Paperscited by 112
  24. 2014Matrix Completion and Low-Rank SVD via Fast Alternating Least Squarescited by 104
  25. 2019Noisy Matrix Completion: Understanding Statistical Guarantees for Convex Relaxation via Nonconvex Optimizationcited by 58
  26. 2019Nonconvex Low-Rank Tensor Completion from Noisy Datacited by 35
  27. 2023Open Problem: Optimal Best Arm Identification with Fixed Budgetcited by 18
  28. 2019PAC Identification of Many Good Arms in Stochastic Multi-Armed Banditscited by 15

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 3; 2026: 11.

14 papers.

Sort:

By area:

  1. 2025How Benchmark Prediction from Fewer Data Misses the Markcited by 26
  2. 2026Efficient Evaluation of LLM Performance with Statistical Guaranteescited by 16
  3. 2025A Multi-Agent Conversational Bandit Approach to Online Evaluation and Selection of User-Aligned LLM Responsescited by 8
  4. 2025Accelerating Unbiased LLM Evaluation via Synthetic Feedbackcited by 6
  5. 2026Towards Reliable LLM Evaluation: Correcting the Winner's Curse in Adaptive Benchmarkingcited by 5
  6. 2026Valid Best-Model Identification for LLM Evaluation via Low-Rank Factorizationcited by 3
  7. 2026Cutting LLM Evaluation Costs with SySRs: A Bandit Algorithm that Provably Exploits Model Similaritycited by 1
  8. 2026You Don't Need to Run Every Evalcited by 1
  9. 2026Cost-Aware Multi-Objective Bandits: Theory and Application to Budgeted LLM Configuration Evaluationcited by 1
  10. 2026AgentOpt v0.1 Technical Report: Client-Side Optimization for LLM-Based Agentcited by 0
  11. 2026LLMSYS-HPOBench: Hyperparameter Optimization Benchmark Suite for Real-World LLM Systemscited by 0
  12. 2026Beyond Independent Labels: Schwartz-Geometry Decoding for Human Value Detectioncited by 0
2 more
  1. 2026Dynamically Allocating Evaluation Effort for Model Rankingcited by 0
  2. 2026Efficient Cost-Aware LLM Evaluation via Bayesian Bandit Gittins Indicescited by 0