mlsanityData to 7 Oct 2026

Adding Error Bars to Evals: A Statistical Approach to Language Model Evaluations

arXiv 2411.00640 ↗First posted 1 Nov 2024stat.AP cs.CLcited by 130

Not on the map. Why?

Cited by 27 in LLM agents and tool use, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

Evaluations are critical for understanding the capabilities of large language models (LLMs). Fundamentally, evaluations are experiments; but the literature on evaluations has largely ignored the literature from other sciences on experiment analysis and planning. This article shows researchers with some training in statistics how to think about and analyze data from language model evaluations. Conceptualizing evaluation questions as having been drawn from an unseen super-population, we present formulas for analyzing evaluation data, measuring differences between two models, and planning an evaluation experiment. We make a number of specific recommendations for running language model evaluations and reporting experiment results in a way that minimizes statistical noise and maximizes informativeness.

Key facts. Cited by 130 papers in the map (data through 7 Oct 2026). Builds on 11 of its 18 references that are in the map. First posted on arXiv 1 Nov 2024.

Builds on

11 of 18 references.

References · 11

  1. 2024The Llama 3 Herd of Modelscited by 14,192
  2. 2015Distilling the Knowledge in a Neural Networkcited by 10,076
  3. 2021Evaluating Large Language Models Trained on Codecited by 8,053
  4. 2021Measuring Mathematical Problem Solving With the MATH Datasetcited by 5,203
  5. 2018Know What You Don't Know: Unanswerable Questions for SQuADcited by 1,538
  6. 2024Chatbot Arena: An Open Platform for Evaluating LLMs by Human Preferencecited by 1,151
  7. 2019DROP: A Reading Comprehension Benchmark Requiring Discrete Reasoning Over Paragraphscited by 883
  8. 2017RACE: Large-scale ReAding Comprehension Dataset From Examinationscited by 851
  9. 2022Language Models are Multilingual Chain-of-Thought Reasonerscited by 518
  10. 2018QuAC : Question Answering in Contextcited by 379
  11. 2024Quantifying Variance in Evaluation Benchmarkscited by 52

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 51; 2026: 79.

130 papers.

Sort:

By area:

  1. 2025Measuring what Matters: Construct Validity in Large Language Model Benchmarkscited by 102
  2. 2025ZeroBench: An Impossible Visual Benchmark for Contemporary Large Multimodal Modelscited by 56
  3. 2025AMO-Bench: Large Language Models Still Struggle in High School Math Competitionscited by 36
  4. 2025ChatBench: From Static Benchmarks to Human-AI Evaluationcited by 31
  5. 2025AgentsNet: Coordination and Collaborative Reasoning in Multi-Agent LLMscited by 31
  6. 2025How to Correctly Report LLM-as-a-Judge Evaluationscited by 25
  7. 2025Position: Don't Use the CLT in LLM Evals With Fewer Than a Few Hundred Datapointscited by 22
  8. 2025Signal and Noise: A Framework for Reducing Uncertainty in Language Model Evaluationcited by 22
  9. 2025LLM-Safety Evaluations Lack Robustnesscited by 16
  10. 2025Literature Review Of Multi-Agent Debate For Problem-Solvingcited by 15
  11. 2025HiBayES: A Hierarchical Bayesian Modeling Framework for AI Evaluation Statisticscited by 14
  12. 2025Do Repetitions Matter? Strengthening Reliability in LLM Evaluationscited by 14
38 more
  1. 2025Prompting Science Report 2: The Decreasing Value of Chain of Thought in Promptingcited by 13
  2. 2026On Randomness in Agentic Evalscited by 12
  3. 2025Thinker: Learning to Think Fast and Slowcited by 10
  4. 2025HumanAgencyBench: Scalable Evaluation of Human Agency Support in AI Assistantscited by 10
  5. 2026Benchmarks Saturate When The Model Gets Smarter Than The Judgecited by 10
  6. 2025What do Large Language Models Say About Animals? Investigating Risks of Animal Harm in Generated Textcited by 9
  7. 2025Cer-Eval: Certifiable and Cost-Efficient Evaluation Framework for LLMscited by 9
  8. 2025LLMs Outperform Experts on Challenging Biology Benchmarkscited by 9
  9. 2025STREAM (ChemBio): A Standard for Transparently Reporting Evaluations in AI Model Reportscited by 9
  10. 2025Stochasticity in Agentic Evaluations: Quantifying Inconsistency with Intraclass Correlationcited by 9
  11. 2025A validity-guided workflow for robust large language model research in psychologycited by 8
  12. 2025Evaluation of Large Language Models via Coupled Token Generationcited by 7
  13. 2025Confidence in Large Language Model Evaluation: A Bayesian Approach to Limited-Sample Challengescited by 7
  14. 2026Vibe Code Bench: Evaluating AI Models on End-to-End Web Application Developmentcited by 7
  15. 2026Hidden Measurement Error in LLM Pipelines Distorts Annotation, Evaluation, and Benchmarkingcited by 7
  16. 2025OIBench: Benchmarking Strong Reasoning Models with Olympiad in Informaticscited by 6
  17. 2025ReasonBENCH: Benchmarking the (In)Stability of LLM Reasoningcited by 6
  18. 2026Resolution Diagnostics for Paired LLM Evaluationcited by 6
  19. 2025Beyond the Singular: Revealing the Value of Multiple Generations in Benchmark Evaluationcited by 5
  20. 2025Identities are not Interchangeable: The Problem of Overgeneralization in Fair Machine Learningcited by 5
  21. 2025Do Large Language Models (Really) Need Statistical Foundations?cited by 5
  22. 2025Measuring all the noises of LLM Evalscited by 5
  23. 2026When LLMs get significantly worse: A statistical approach to detect model degradationscited by 5
  24. 2026Rank Intervals for Leaderboards: A Hierarchical Framework for Model Evaluationcited by 5
  25. 2026DarwinX: Evolving Agent Harnesses Through Natural Selectioncited by 5
  26. 2025Position: Ensuring mutual privacy is necessary for effective external evaluation of proprietary AI systemscited by 4
  27. 2025Statistical Methods in Generative AIcited by 4
  28. 2026CyclicJudge: Mitigating Judge Bias Efficiently in LLM-based Evaluationcited by 4
  29. 2026Beyond the Mean: Within-Model Reliable Change Detection for LLM Evaluationcited by 4
  30. 2026Bias and Uncertainty in LLM-as-a-Judge Estimationcited by 4
  31. 2026Expert Evaluation of Clinical AI Tools on Real Point-of-Care Clinical Queriescited by 4
  32. 2026Test-Time Scaling in Reasoning LLMs: Inference Regimes, Evaluation, and Reproducibilitycited by 4
  33. 2025Developing and Maintaining an Open-Source Repository of AI Evaluations: Challenges and Insightscited by 3
  34. 2025LLM-as-a-Judge: Rapid Evaluation of Legal Document Recommendation for Retrieval-Augmented Generationcited by 3
  35. 2025Representing LLMs in Prompt Semantic Task Spacecited by 3
  36. 2025LAG-MMLU: Benchmarking Frontier LLM Understanding in Latvian and Giriamacited by 2
  37. 2025Estimating the Self-Consistency of LLMscited by 2
  38. 2025Reproducibility: The New Frontier in AI Governancecited by 2

50 most cited of 130 All 130 (JSON)