mlsanityData to 14 Oct 2026

tinyBenchmarks: evaluating LLMs with fewer examples

arXiv 2402.14992 ↗First posted 22 Feb 2024cs.CL cs.AI cs.LGcited by 310

Not on the map. Why?

Cited by 26 in LLM agents and tool use, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

The versatility of large language models (LLMs) led to the creation of diverse benchmarks that thoroughly test a variety of language models' abilities. These benchmarks consist of tens of thousands of examples making evaluation of LLMs very expensive. In this paper, we investigate strategies to reduce the number of evaluations needed to assess the performance of an LLM on several key benchmarks. For example, we show that to accurately estimate the performance of an LLM on MMLU, a popular multiple-choice QA benchmark consisting of 14K examples, it is sufficient to evaluate this LLM on 100 curated examples. We release evaluation tools and tiny versions of popular benchmarks: Open LLM Leaderboard, MMLU, HELM, and AlpacaEval 2.0. Our empirical analysis demonstrates that these tools and tiny benchmarks are sufficient to reliably and efficiently reproduce the original evaluation results.

Key facts. Cited by 310 papers in the map (data through 14 Oct 2026). Builds on 38 of its 62 references that are in the map. First posted on arXiv 22 Feb 2024.

Builds on

38 of 62 references.

References · 38

  1. 2020Language Models are Few-Shot Learnerscited by 32,419
  2. 2023GPT-4 Technical Reportcited by 18,396
  3. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 13,028
  4. 2021Training Verifiers to Solve Math Word Problemscited by 10,109
  5. 2020Measuring Massive Multitask Language Understandingcited by 7,963
  6. 2021Measuring Mathematical Problem Solving With the MATH Datasetcited by 5,371
  7. 2018GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understandingcited by 4,899
  8. 2018Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challengecited by 4,886
  9. 2019HellaSwag: Can a Machine Really Finish Your Sentence?cited by 4,100
  10. 2021TruthfulQA: Measuring How Models Mimic Human Falsehoodscited by 2,975
  11. 2018Can a Suit of Armor Conduct Electricity? A New Dataset for Open Book Question Answeringcited by 2,164
  12. 2022Beyond the Imitation Game: Quantifying and extrapolating the capabilities of language modelscited by 1,998
26 more
  1. 2022Rethinking the Role of Demonstrations: What Makes In-Context Learning Work?cited by 1,434
  2. 2020What Disease does this Patient Have? A Large-scale Open Domain Question Answering Dataset from Medical Examscited by 1,356
  3. 2021Fantastically Ordered Prompts and Where to Find Them: Overcoming Few-Shot Prompt Order Sensitivitycited by 1,135
  4. 2023Quantifying Language Models' Sensitivity to Spurious Features in Prompt Design or: How I learned to start worrying about prompt formattingcited by 817
  5. 2019Adversarial NLI: A New Benchmark for Natural Language Understandingcited by 807
  6. 2017The NarrativeQA Reading Comprehension Challengecited by 597
  7. 2023Larger language models do in-context learning differentlycited by 394
  8. 2023LegalBench: A Collaboratively Built Benchmark for Measuring Legal Reasoning in Large Language Modelscited by 336
  9. 2023State of What Art? A Call for Multi-Prompt LLM Evaluationcited by 310
  10. 2022PromptSource: An Integrated Development Environment and Repository for Natural Language Promptscited by 289
  11. 2021Reframing Instructional Prompts to GPTk's Languagecited by 176
  12. 2023Emergent and Predictable Memorization in Large Language Modelscited by 148
  13. 2022Ground-Truth Labels Matter: A Deeper Look into Input-Label Demonstrationscited by 116
  14. 2023LLM360: Towards Fully Transparent Open-Source LLMscited by 104
  15. 2023Anchor Points: Benchmarking Models with Much Fewer Examplescited by 86
  16. 2024Mind Your Format: Towards Consistent Evaluation of In-Context Learning Improvementscited by 83
  17. 2021Active Testing: Sample-Efficient Model Evaluationcited by 82
  18. 2023Efficient Benchmarking of Language Modelscited by 71
  19. 2021Comparing Test Sets with Item Response Theorycited by 57
  20. 2023Position: AI Evaluation Should Learn from How We Test Humanscited by 36
  21. 2023How Predictable Are Large Language Model Capabilities? A Case Study on BIG-benchcited by 34
  22. 2023Mind the instructions: a holistic evaluation of consistency and interactions in prompt-based learningcited by 31
  23. 2022py-irt: A Scalable Item Response Theory Library for Pythoncited by 19
  24. 2020Active Bayesian Assessment for Black-Box Classifierscited by 17
  25. 2020Effective Sample Size, Dimensionality, and Generalization in Covariate Shift Adaptationcited by 13
  26. 2023The ICL Consistency Testcited by 6

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 52; 2025: 84; 2026: 174.

310 papers.

Sort:

By area:

  1. 2024Refusal in Language Models Is Mediated by a Single Directioncited by 1,032
  2. 2024LMMs-Eval: Reality Check on the Evaluation of Large Multimodal Modelscited by 353
  3. 2025Small Language Models are the Future of Agentic AIcited by 270
  4. 2024AssistantBench: Can Web Agents Solve Realistic and Time-Consuming Tasks?cited by 158
  5. 2024Observational Scaling Laws and the Predictability of Language Model Performancecited by 130
  6. 2025Measuring what Matters: Construct Validity in Large Language Model Benchmarkscited by 119
  7. 2025SealQA: Raising the Bar for Reasoning in Search-Augmented Language Modelscited by 75
  8. 2024Efficient multi-prompt evaluation of LLMscited by 73
  9. 2024Quantifying Variance in Evaluation Benchmarkscited by 54
  10. 2025Angular Steering: Behavior Control via Rotation in Activation Spacecited by 53
  11. 2024SUPER: Evaluating Agents on Setting Up and Executing Tasks from Research Repositoriescited by 52
  12. 2025Which of These Best Describes Multiple Choice Evaluation with LLMs? A) Forced B) Flawed C) Fixable D) All of the Abovecited by 52
38 more
  1. 2024CiteME: Can Language Models Accurately Cite Scientific Claims?cited by 48
  2. 2024AutoBencher: Towards Declarative Benchmark Constructioncited by 44
  3. 2024On the Worst Prompt Performance of Large Language Modelscited by 38
  4. 2024NLSR: Neuron-Level Safety Realignment of Large Language Models Against Harmful Fine-Tuningcited by 37
  5. 2025A Rosetta Stone for AI Benchmarkscited by 36
  6. 2026Group-Evolving Agents: Open-Ended Self-Improvement via Experience Sharingcited by 36
  7. 2024Benchmarks as Microscopes: A Call for Model Metrologycited by 35
  8. 2024Active Evaluation Acquisition for Efficient LLM Benchmarkingcited by 35
  9. 2024Mechanistic Unlearning: Robust Knowledge Unlearning and Editing via Mechanistic Localizationcited by 35
  10. 2024Surveying the Effects of Quality, Diversity, and Complexity in Synthetic Data From Large Language Modelscited by 35
  11. 2025Fluid Language Model Benchmarkingcited by 35
  12. 2024Are large language models superhuman chemists?cited by 34
  13. 2024Examining the robustness of LLM evaluation to the distributional assumptions of benchmarkscited by 31
  14. 2025Reliable and Efficient Amortized Model-based Evaluationcited by 31
  15. 2025How Benchmark Prediction from Fewer Data Misses the Markcited by 28
  16. 2025Adaptive Testing for LLM Evaluation: A Psychometric Alternative to Static Benchmarkscited by 28
  17. 2024Do These LLM Benchmarks Agree? Fixing Benchmark Evaluation with BenchBenchcited by 27
  18. 2024Distributional Preference Alignment of LLMs via Optimal Transportcited by 26
  19. 2025Signal and Noise: A Framework for Reducing Uncertainty in Language Model Evaluationcited by 26
  20. 2024100 instances is all you need: predicting the success of a new LLM on unseen data by testing on a few instancescited by 24
  21. 2024Noise Injection Reveals Hidden Capabilities of Sandbagging Language Modelscited by 24
  22. 2024Sloth: scaling laws for LLM skills to predict multi-benchmark performance across familiescited by 24
  23. 2025How to Select Datapoints for Efficient Human Evaluation of NLG Models?cited by 23
  24. 2025Steering Language Models with Weight Arithmeticcited by 20
  25. 2025LLM-Safety Evaluations Lack Robustnesscited by 18
  26. 2026Efficient Evaluation of LLM Performance with Statistical Guaranteescited by 18
  27. 2024Raising the Bar: Investigating the Values of Large Language Models via Generative Evolving Testingcited by 17
  28. 2024LiveXiv -- A Multi-Modal Live Benchmark Based on Arxiv Papers Contentcited by 17
  29. 2025Personality Vector: Modulating Personality of Large Language Models by Model Mergingcited by 17
  30. 2024ONEBench to Test Them All: Sample-Level Benchmarking Over Open-Ended Capabilitiescited by 15
  31. 2025Activation Steering with a Feedback Controllercited by 15
  32. 2024A transfer learning framework for weak-to-strong generalizationcited by 14
  33. 2025Scalable Fingerprinting of Large Language Modelscited by 14
  34. 2025Scaling Up Active Testing to Large Language Modelscited by 13
  35. 2024Privacy-Aware Visual Language Modelscited by 12
  36. 2024Data Efficient Evaluation of Large Language Models and Text-to-Image Models via Adaptive Samplingcited by 12
  37. 2024Badllama 3: removing safety finetuning from Llama 3 in minutescited by 11
  38. 2025Paradigms of AI Evaluation: Mapping Goals, Methodologies and Culturecited by 11

50 most cited of 310 All 310 (JSON)