# Examining the robustness of LLM evaluation to the distributional assumptions of benchmarks

arXiv: https://arxiv.org/abs/2404.16966 · HTML: https://mlsanity.com/paper/2404.16966/

## Key facts

- Cited by 31 papers in the map (data through 14 Oct 2026).
- Builds on 38 of its 50 references that are in the map.
- First posted on arXiv 25 Apr 2024.

## Builds on (most cited first, 30 of 38)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [LLaMA: Open and Efficient Foundation Language Models](https://mlsanity.com/paper/2302.13971/) (2023)
- [HellaSwag: Can a Machine Really Finish Your Sentence?](https://mlsanity.com/paper/1905.07830/) (2019)
- [Finetuned Language Models Are Zero-Shot Learners](https://mlsanity.com/paper/2109.01652/) (2021)
- [Making the V in VQA Matter: Elevating the Role of Image Understanding in Visual Question Answering](https://mlsanity.com/paper/1612.00837/) (2016)
- [CommonsenseQA: A Question Answering Challenge Targeting Commonsense Knowledge](https://mlsanity.com/paper/1811.00937/) (2018)
- [Get To The Point: Summarization with Pointer-Generator Networks](https://mlsanity.com/paper/1704.04368/) (2017)
- [GLM-130B: An Open Bilingual Pre-trained Model](https://mlsanity.com/paper/2210.02414/) (2022)
- [Quantifying Language Models' Sensitivity to Spurious Features in Prompt Design or: How I learned to start worrying about prompt formatting](https://mlsanity.com/paper/2310.11324/) (2023)
- [Abstractive Text Summarization Using Sequence-to-Sequence RNNs and Beyond](https://mlsanity.com/paper/1602.06023/) (2016)
- [Adversarial NLI: A New Benchmark for Natural Language Understanding](https://mlsanity.com/paper/1910.14599/) (2019)
- [Generalizing to Unseen Domains: A Survey on Domain Generalization](https://mlsanity.com/paper/2103.03097/) (2021)
- [Annotation Artifacts in Natural Language Inference Data](https://mlsanity.com/paper/1803.02324/) (2018)
- [Automatic Prompt Optimization with "Gradient Descent" and Beam Search](https://mlsanity.com/paper/2305.03495/) (2023)
- [How Far Can Camels Go? Exploring the State of Instruction Tuning on Open Resources](https://mlsanity.com/paper/2306.04751/) (2023)
- [RLPrompt: Optimizing Discrete Text Prompts with Reinforcement Learning](https://mlsanity.com/paper/2205.12548/) (2022)
- [State of What Art? A Call for Multi-Prompt LLM Evaluation](https://mlsanity.com/paper/2401.00595/) (2023)
- [tinyBenchmarks: evaluating LLMs with fewer examples](https://mlsanity.com/paper/2402.14992/) (2024)
- [Proving Test Set Contamination in Black Box Language Models](https://mlsanity.com/paper/2310.17623/) (2023)
- [PromptRobust: Towards Evaluating the Robustness of Large Language Models on Adversarial Prompts](https://mlsanity.com/paper/2306.04528/) (2023)
- [Probing Neural Network Comprehension of Natural Language Arguments](https://mlsanity.com/paper/1907.07355/) (2019)
- [On the Robustness of ChatGPT: An Adversarial and Out-of-distribution Perspective](https://mlsanity.com/paper/2302.12095/) (2023)
- [Evaluating Large Language Models: A Comprehensive Survey](https://mlsanity.com/paper/2310.19736/) (2023)
- [Don't Make Your LLM an Evaluation Benchmark Cheater](https://mlsanity.com/paper/2311.01964/) (2023)
- [Adversarial Filters of Dataset Biases](https://mlsanity.com/paper/2002.04108/) (2020)
- [When Benchmarks are Targets: Revealing the Sensitivity of Large Language Model Leaderboards](https://mlsanity.com/paper/2402.01781/) (2024)
- [Robustness Gym: Unifying the NLP Evaluation Landscape](https://mlsanity.com/paper/2101.04840/) (2021)
- [Measure and Improve Robustness in NLP Models: A Survey](https://mlsanity.com/paper/2112.08313/) (2021)
- [Anchor Points: Benchmarking Models with Much Fewer Examples](https://mlsanity.com/paper/2309.08638/) (2023)

## Built on by (most cited first, 30 of 31)

- [On-Device Language Models: A Comprehensive Review](https://mlsanity.com/paper/2409.00088/) (2024)
- [What Did I Do Wrong? Quantifying LLMs' Sensitivity and Consistency to Prompt Engineering](https://mlsanity.com/paper/2406.12334/) (2024)
- [100 instances is all you need: predicting the success of a new LLM on unseen data by testing on a few instances](https://mlsanity.com/paper/2409.03563/) (2024)
- [AgentNoiseBench: Benchmarking Robustness of Tool-Using LLM Agents Under Noisy Condition](https://mlsanity.com/paper/2602.11348/) (2026)
- [Towards Acyclic Preference Evaluation of Language Models via Multiple Evaluators](https://mlsanity.com/paper/2410.12869/) (2024)
- [Robustness in Large Language Models: A Survey of Mitigation Strategies and Evaluation Metrics](https://mlsanity.com/paper/2505.18658/) (2025)
- [ReliableEval: A Recipe for Stochastic LLM Evaluation via Method of Moments](https://mlsanity.com/paper/2505.22169/) (2025)
- [SelfPrompt: Autonomously Evaluating LLM Robustness via Domain-Constrained Knowledge Guidelines and Refined Adversarial Prompts](https://mlsanity.com/paper/2412.00765/) (2024)
- [Shortcut Learning in In-Context Learning: A Survey](https://mlsanity.com/paper/2411.02018/) (2024)
- [ErrorMap and ErrorAtlas: Charting the Failure Landscape of Large Language Models](https://mlsanity.com/paper/2601.15812/) (2026)
- [Learning to Act under Noise: Enhancing Agent Robustness via Noisy Environments](https://mlsanity.com/paper/2605.27209/) (2026)
- [How Many Tasks Are Enough for Agent Benchmark Decisions? A Replay Analysis of Public LLM Agent Benchmarks](https://mlsanity.com/paper/2607.12338/) (2026)
- [Can We Trust Item Response Theory for AI Evaluation?](https://mlsanity.com/paper/2607.15190/) (2026)
- [KGPA: Robustness Evaluation for Large Language Models via Cross-Domain Knowledge Graphs](https://mlsanity.com/paper/2406.10802/) (2024)
- [Adapting Large Language Models for Multi-Domain Retrieval-Augmented-Generation](https://mlsanity.com/paper/2504.02411/) (2025)
- [Automated Capability Evaluation of Foundation Models](https://mlsanity.com/paper/2505.17228/) (2025)
- [A Principle-based Framework for the Development and Evaluation of Large Language Models for Health and Wellness](https://mlsanity.com/paper/2512.08936/) (2025)
- [Interactive Evaluation Requires a Design Science](https://mlsanity.com/paper/2605.17829/) (2026)
- [Exploring Explanations Improves the Robustness of In-Context Learning](https://mlsanity.com/paper/2506.02378/) (2025)
- [LLM Benchmark Datasets Should Be Contamination-Resistant](https://mlsanity.com/paper/2605.19999/) (2026)
- [Robust AI Evaluation through Maximal Lotteries](https://mlsanity.com/paper/2602.21297/) (2026)
- [Towards Effective In-context Cross-domain Knowledge Transfer via Domain-invariant-neurons-based Retrieval](https://mlsanity.com/paper/2604.05383/) (2026)
- [Reason Analogically via Cross-domain Prior Knowledge: An Empirical Study of Cross-domain Knowledge Transfer for In-Context Learning](https://mlsanity.com/paper/2604.05396/) (2026)
- [Are Near-Tied LLM Rankings Robust to Family-DIF-Guided Benchmark Recomposition?](https://mlsanity.com/paper/2609.00482/) (2026)
- [Surfacing Semantic Orthogonality Across Model Safety Benchmarks: A Multi-Dimensional Analysis](https://mlsanity.com/paper/2505.17636/) (2025)
- [Evaluating Prompt Engineering Techniques for Accuracy and Confidence Elicitation in Medical LLMs](https://mlsanity.com/paper/2506.00072/) (2025)
- [Performance Assessment Strategies for Language Model Applications in Healthcare](https://mlsanity.com/paper/2509.08087/) (2025)
- [Correcting Prompt Dependence in LLM Benchmarks: A Bayesian Hierarchical Model with Embedding-Space Clustering](https://mlsanity.com/paper/2510.05709/) (2025)
- [CoDA: Towards Effective Cross-domain Knowledge Transfer via CoT-guided Domain Adaptation](https://mlsanity.com/paper/2604.19488/) (2026)
- [When Efficiency Backfires: Cascading LLMs Trigger Cascade Failure under Adversarial Attack](https://mlsanity.com/paper/2605.17288/) (2026)

## Abstract (arXiv)

Benchmarks have emerged as the central approach for evaluating Large Language Models (LLMs). The research community often relies on a model's average performance across the test prompts of a benchmark to evaluate the model's performance. This is consistent with the assumption that the test prompts within a benchmark represent a random sample from a real-world distribution of interest. We note that this is generally not the case; instead, we hold that the distribution of interest varies according to the specific use case. We find that (1) the correlation in model performance across test prompts is non-random, (2) accounting for correlations across test prompts can change model rankings on major benchmarks, (3) explanatory factors for these correlations include semantic similarity and common LLM failure points.