mlsanityData to 7 Oct 2026

The Good, The Bad, and The Greedy: Evaluation of LLMs Should Not Ignore Non-Determinism

arXiv 2407.10457 ↗First posted 15 Jul 2024cs.CL cs.AIcited by 113

Not on the map. Why?

Abstract (arXiv)

Current evaluations of large language models (LLMs) often overlook non-determinism, typically focusing on a single output per example. This limits our understanding of LLM performance variability in real-world applications. Our study addresses this issue by exploring key questions about the performance differences between greedy decoding and sampling, identifying benchmarks' consistency regarding non-determinism, and examining unique model behaviors. Through extensive experiments, we observe that greedy decoding generally outperforms sampling methods for most evaluated tasks. We also observe consistent performance across different LLM sizes and alignment methods, noting that alignment can reduce sampling variance. Moreover, our best-of-N sampling approach demonstrates that smaller LLMs can match or surpass larger models such as GPT-4-Turbo, highlighting the untapped potential of smaller LLMs. This research shows the importance of considering non-determinism in LLM evaluations and provides insights for future LLM development and evaluation.

Key facts. Cited by 113 papers in the map (data through 7 Oct 2026). Builds on 35 of its 54 references that are in the map. First posted on arXiv 15 Jul 2024.

Builds on

35 of 54 references.

References · 35

  1. 2021Training Verifiers to Solve Math Word Problemscited by 9,743
  2. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,393
  3. 2021Evaluating Large Language Models Trained on Codecited by 8,053
  4. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 7,916
  5. 2020Measuring Massive Multitask Language Understandingcited by 7,768
  6. 2022Self-Consistency Improves Chain of Thought Reasoning in Language Modelscited by 5,601
  7. 2021Measuring Mathematical Problem Solving With the MATH Datasetcited by 5,203
  8. 2018Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challengecited by 4,710
  9. 2019HellaSwag: Can a Machine Really Finish Your Sentence?cited by 4,018
  10. 2021Program Synthesis with Large Language Modelscited by 3,509
  11. 2019The Curious Case of Neural Text Degenerationcited by 2,881
  12. 2024MMLU-Pro: A More Robust and Challenging Multi-Task Language Understanding Benchmarkcited by 1,758
23 more
  1. 2018Hierarchical Neural Story Generationcited by 1,182
  2. 2024SimPO: Simple Preference Optimization with a Reference-Free Rewardcited by 951
  3. 2023A General Theoretical Paradigm to Understand Learning from Human Preferencescited by 906
  4. 2023Quantifying Language Models' Sensitivity to Spurious Features in Prompt Design or: How I learned to start worrying about prompt formattingcited by 778
  5. 2019CTRL: A Conditional Transformer Language Model for Controllable Generationcited by 754
  6. 2023LLM-Blender: Ensembling Large Language Models with Pairwise Ranking and Generative Fusioncited by 550
  7. 2024RewardBench: Evaluating Reward Models for Language Modelingcited by 480
  8. 2024Improve Mathematical Reasoning in Language Models by Automated Process Supervisioncited by 426
  9. 2023How Far Can Camels Go? Exploring the State of Instruction Tuning on Open Resourcescited by 405
  10. 2024Interpretable Preferences via Multi-Objective Reward Modeling and Mixture-of-Expertscited by 369
  11. 2023Iterative Preference Learning from Human Feedback: Bridging Theory and Practice for RLHF under KL-Constraintcited by 346
  12. 2023State of What Art? A Call for Multi-Prompt LLM Evaluationcited by 299
  13. 2017Beam Search Strategies for Neural Machine Translationcited by 232
  14. 2024WildBench: Benchmarking LLMs with Challenging Tasks from Real Users in the Wildcited by 211
  15. 2024Disentangling Length from Quality in Direct Preference Optimizationcited by 194
  16. 2023SwiftSage: A Generative Agent with Fast and Slow Thinking for Complex Interactive Taskscited by 166
  17. 2024AlphaMath Almost Zero: Process Supervision without Processcited by 160
  18. 2024Accessing GPT-4 level Mathematical Olympiad Solutions via Monte Carlo Tree Self-refine with LLaMa-3 8Bcited by 156
  19. 2024When Benchmarks are Targets: Revealing the Sensitivity of Large Language Model Leaderboardscited by 134
  20. 2024Common 7B Language Models Already Possess Strong Math Capabilitiescited by 108
  21. 2024Watch Every Step! LLM Agent Learning via Iterative Step-Level Process Refinementcited by 86
  22. 2024MixEval: Deriving Wisdom of the Crowd from LLM Benchmark Mixturescited by 73
  23. 2024The Larger the Better? Improved LLM Code-Generation via Budget Reallocationcited by 41

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 9; 2025: 61; 2026: 43.

113 papers.

Sort:

By area:

  1. 2024Large Language Monkeys: Scaling Inference Compute with Repeated Samplingcited by 946
  2. 2025A Survey on Test-Time Scaling in Large Language Models: What, How, Where, and How Well?cited by 215
  3. 2025Light-R1: Curriculum SFT, DPO and RL for Long COT from Scratch and Beyondcited by 175
  4. 2024Non-Determinism of "Deterministic" LLM Settingscited by 84
  5. 2025NoveltyBench: Evaluating Language Models for Humanlike Diversitycited by 64
  6. 2024Smaller, Weaker, Yet Better: Training LLM Reasoners via Compute-Optimal Samplingcited by 60
  7. 2025RoboMonkey: Scaling Test-Time Sampling and Verification for Vision-Language-Action Modelscited by 56
  8. 2024The Limits of Inference Scaling Through Resamplingcited by 51
  9. 2025Base Models Beat Aligned Models at Randomness and Creativitycited by 48
  10. 2024HelloBench: Evaluating Long Text Generation Capabilities of Large Language Modelscited by 42
  11. 2024A Comprehensive Evaluation of Cognitive Biases in LLMscited by 32
  12. 2025LLM Probability Concentration: How Alignment Shrinks the Generative Horizoncited by 26
38 more
  1. 2025LogicAD: Explainable Anomaly Detection via VLM-based Text Feature Extractioncited by 24
  2. 2024Repetition Neurons: How Do Language Models Produce Repetitions?cited by 22
  3. 2025Verifier-free Test-Time Sampling for Vision-Language-Action Modelscited by 21
  4. 2024Better Instruction-Following Through Minimum Bayes Riskcited by 19
  5. 2025RobustFlow: Towards Robust Agentic Workflow Generationcited by 19
  6. 2025Argos: Agentic Time-Series Anomaly Detection with Autonomous Rule Generation via Large Language Modelscited by 17
  7. 2025Minions: Cost-efficient Collaboration Between On-device and Cloud Language Modelscited by 17
  8. 2025Learning from Peers in Reasoning Modelscited by 16
  9. 2025LIMOPro: Reasoning Refinement for Efficient and Effective Test-time Scalingcited by 15
  10. 2026LLM-42: Enabling Determinism in LLM Inference with Verified Speculationcited by 14
  11. 2025Sample Complexity and Representation Ability of Test-time Scaling Paradigmscited by 12
  12. 2025Query-Level Uncertainty in Large Language Modelscited by 12
  13. 2025Certified Self-Consistency: Statistical Guarantees and Test-Time Training for Reliable Reasoning in LLMscited by 12
  14. 2025Reflections on the Reproducibility of Commercial LLM Performance in Empirical Software Engineering Studiescited by 12
  15. 2025ReliableEval: A Recipe for Stochastic LLM Evaluation via Method of Momentscited by 10
  16. 2025INTER: Mitigating Hallucination in Large Vision-Language Models by Interaction Guidance Samplingcited by 10
  17. 2025Reasoning Path Divergence: A New Metric and Curation Strategy to Unlock LLM Diverse Thinkingcited by 10
  18. 2025Repetitions are not all alike: distinct mechanisms sustain repetition in language modelscited by 9
  19. 2025Do LLMs Understand Your Translations? Evaluating Paragraph-level MT with Question Answeringcited by 9
  20. 2025Fluent but Foreign: Even Regional LLMs Lack Cultural Alignmentcited by 9
  21. 2025When Life Gives You Samples: The Benefits of Scaling up Inference Compute for Multilingual LLMscited by 9
  22. 2025Failure Modes in LLM Systems: A System-Level Taxonomy for Reliable AI Applicationscited by 9
  23. 2026ToolFailBench: Diagnosing Tool-Use Failures in LLM Agentscited by 8
  24. 2025Critical Foreign Policy Decisions (CFPD)-Benchmark: Measuring Diplomatic Preferences in Large Language Modelscited by 7
  25. 2025ATLAS: Adaptive Trading with LLM AgentS Through Dynamic Prompt Optimization and Multi-Agent Coordinationcited by 7
  26. 2025On the Limits of Test-Time Compute: Sequential Reward Filtering for Better Inferencecited by 6
  27. 2025Beyond the Singular: Revealing the Value of Multiple Generations in Benchmark Evaluationcited by 5
  28. 2025Large Language Models Are Unreliable for Cyber Threat Intelligencecited by 5
  29. 2025Mind the Generation Process: Fine-Grained Confidence Estimation During LLM Generationcited by 5
  30. 2026Verified Detection and Prevention of Concurrency Anomalies in Multi-Agent Large Language Model Systemscited by 5
  31. 2025When Harry Meets Superman: The Role of The Interlocutor in Persona-Based Dialogue Generationcited by 4
  32. 2025DecoRTL: A Run-time Decoding Framework for RTL Code Generation with LLMscited by 4
  33. 2025Flow Autoencoders are Effective Protein Tokenizerscited by 4
  34. 2026Reasoning-Driven Synthetic Data Generation and Evaluationcited by 4
  35. 2025Deep-Bench: Deep Learning Benchmark Dataset for Code Generationcited by 3
  36. 2025The Non-Determinism of Small LLMs: Evidence of Low Answer Consistency in Repetition Trials of Standard Multiple-Choice Benchmarkscited by 3
  37. 2025Evaluating Large Language Models for Detecting Antisemitismcited by 3
  38. 2026Assessing Risks of Large Language Models in Mental Health Support: A Framework for Automated Clinical AI Red Teamingcited by 3

50 most cited of 113 All 113 (JSON)