mlsanityData to 7 Oct 2026

Towards Reproducible LLM Evaluation: Quantifying Uncertainty in LLM Benchmark Scores

arXiv 2410.03492 ↗First posted 4 Oct 2024cs.CLcited by 25

Not on the map. Why?

Abstract (arXiv)

Large language models (LLMs) are stochastic, and not all models give deterministic answers, even when setting temperature to zero with a fixed random seed. However, few benchmark studies attempt to quantify uncertainty, partly due to the time and cost of repeated experiments. We use benchmarks designed for testing LLMs' capacity to reason about cardinal directions to explore the impact of experimental repeats on mean score and prediction interval. We suggest a simple method for cost-effectively quantifying the uncertainty of a benchmark score and make recommendations concerning reproducible LLM evaluation.

Key facts. Cited by 25 papers in the map (data through 7 Oct 2026). Builds on 7 of its 18 references that are in the map. First posted on arXiv 4 Oct 2024.

Builds on

7 of 18 references.

References · 7

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  2. 2020Language Models are Few-Shot Learnerscited by 32,199
  3. 2019The Curious Case of Neural Text Degenerationcited by 2,881
  4. 2024BigCodeBench: Benchmarking Code Generation with Diverse Function Calls and Complex Instructionscited by 527
  5. 2023LegalBench: A Collaboratively Built Benchmark for Measuring Legal Reasoning in Large Language Modelscited by 328
  6. 2024Evaluating the Ability of Large Language Models to Reason about Cardinal Directionscited by 10
  7. 2023Towards Inferential Reproducibility of Machine Learning Researchcited by 4

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 12; 2026: 13.

25 papers.

Sort:

By area:

  1. 2025Foundation Models for Geospatial Reasoning: Assessing Capabilities of Large Language Models in Understanding Geometries and Topological Spatial Relationscited by 20
  2. 2025LLMs Provide Unstable Answers to Legal Questionscited by 19
  3. 2025Reflections on the Reproducibility of Commercial LLM Performance in Empirical Software Engineering Studiescited by 12
  4. 2025Confidence in Large Language Model Evaluation: A Bayesian Approach to Limited-Sample Challengescited by 7
  5. 2025ReasonBENCH: Benchmarking the (In)Stability of LLM Reasoningcited by 6
  6. 2026The Silent Hyperparameter: Quantifying the Impact of Inference Backends on LLM Reproducibilitycited by 6
  7. 2026Test-Time Scaling in Reasoning LLMs: Inference Regimes, Evaluation, and Reproducibilitycited by 4
  8. 2026Beyond Reproducibility: Token Probabilities Expose Large Language Model Nondeterminismcited by 3
  9. 2025Toward Robust Legal Text Formalization into Defeasible Deontic Logic using LLMscited by 2
  10. 2025From Queries to Criteria: Understanding How Astronomers Evaluate LLMscited by 2
  11. 2025Investigating LLM Variability in Personalized Conversational Information Retrievalcited by 2
  12. 2025Critical Confabulation: Can LLMs Hallucinate for Social Good?cited by 2
13 more
  1. 2025Evaluating the Ability of Large Language Models to Reason about Cardinal Directions, Revisitedcited by 1
  2. 2025DecompSR: A dataset for decomposed analyses of compositional multihop spatial reasoningcited by 1
  3. 2026Inspectable AI for Science: A Research Object Approach to Generative AI Governancecited by 1
  4. 2026Runtime Governance for Agentic AI: Action-Boundary Control with Trusted Provenance and Fail-Closed Executioncited by 1
  5. 2025Beyond statistical significance: Quantifying uncertainty and statistical variability in multilingual and multitask NLP evaluationcited by 0
  6. 2026AI Agent for Reverse-Engineering Legacy Finite-Difference Code and Translating to Devitocited by 0
  7. 2026CreditAudit: 2nd Dimension for LLM Evaluation and Selectioncited by 0
  8. 2026QSTRBench: a New Benchmark to Evaluate the Ability of Language Models to Reason with Qualitative Spatial and Temporal Calculicited by 0
  9. 2026Accuracy and Cost Claims Do Not Survive Re-Execution in Agentic VideoQAcited by 0
  10. 2026Testing and Evaluation of Agentic AI Systems In Military Command and Controlcited by 0
  11. 2026From Ideas to Actions: A Public-Data Decision-Support Toolchain Across the Venture Lifecyclecited by 0
  12. 2026Speculative Evaluation of Stochastic LLMscited by 0
  13. 2026Reproducible LLM Inference Benchmarking: A Sequential Isolation Protocol for Regression Testingcited by 0