mlsanityData to 7 Oct 2026

The Effect of Sampling Temperature on Problem Solving in Large Language Models

arXiv 2402.05201 ↗First posted 7 Feb 2024cs.CL cs.AIcited by 168

Not on the map. Why?

Cited by 33 in LLM reasoning, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

In this research study, we empirically investigate the effect of sampling temperature on the performance of Large Language Models (LLMs) on various problem-solving tasks. We created a multiple-choice question-and-answer (MCQA) exam by randomly sampling problems from standard LLM benchmarks. Then, we used nine popular LLMs with five prompt-engineering techniques to solve the MCQA problems while increasing the sampling temperature from 0.0 to 1.6. Despite anecdotal reports to the contrary, our empirical results indicate that changes in temperature from 0.0 to 1.0 do not have a statistically significant impact on LLM performance for problem-solving tasks. In addition, these results appear to generalize across LLMs, prompt-engineering techniques, and problem domains. All code, data, and supplemental materials are available on GitHub at: https://github.com/matthewrenze/jhu-llm-temperature

Key facts. Cited by 168 papers in the map (data through 7 Oct 2026). Builds on 19 of its 53 references that are in the map. First posted on arXiv 7 Feb 2024.

Builds on

19 of 53 references.

References · 19

  1. 2023GPT-4 Technical Reportcited by 18,132
  2. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  3. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,302
  4. 2015Distilling the Knowledge in a Neural Networkcited by 10,076
  5. 2019Sentence-BERT: Sentence Embeddings using Siamese BERT-Networkscited by 9,049
  6. 2018Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challengecited by 4,710
  7. 2022Large Language Models are Zero-Shot Reasonerscited by 4,323
  8. 2019HellaSwag: Can a Machine Really Finish Your Sentence?cited by 4,018
  9. 2023AGIEval: A Human-Centric Benchmark for Evaluating Foundation Modelscited by 701
  10. 2023A Prompt Pattern Catalog to Enhance Prompt Engineering with ChatGPTcited by 612
  11. 2022MedMCQA : A Large-scale Multi-Subject Multi-Choice Dataset for Medical domain Question Answeringcited by 525
  12. 2023Augmented Language Models: a Surveycited by 398
7 more
  1. 2022A Systematic Evaluation of Large Language Models of Codecited by 360
  2. 2022Recitation-Augmented Language Modelscited by 65
  3. 2021From LSAT: The Progress and Challenges of Complex Reasoningcited by 43
  4. 2023Cost-Effective Hyperparameter Optimization for Large Language Model Generation Inferencecited by 40
  5. 2023Retrieving Supporting Evidence for Generative Question Answeringcited by 31
  6. 2020Contextual Temperature for Language Modelingcited by 25
  7. 2023Performance of ChatGPT on the US Fundamentals of Engineering Exam: Comprehensive Assessment of Proficiency and Potential Implications for Professional Environmental Engineering Practicecited by 15

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 20; 2025: 88; 2026: 60.

168 papers.

Sort:

By area:

  1. 2025SimpleVLA-RL: Scaling VLA Training via Reinforcement Learningcited by 141
  2. 2025Trae Agent: An LLM-based Agent for Software Engineering with Test-time Scalingcited by 93
  3. 2025A Sober Look at Progress in Language Model Reasoning: Pitfalls and Paths to Reproducibilitycited by 89
  4. 2024A Survey of Text-to-SQL in the Era of LLMs: Where are we, and where are we going?cited by 54
  5. 2025Towards Robust Agentic CUDA Kernel Benchmarking, Verification, and Optimizationcited by 52
  6. 2024Large Language Models have Intrinsic Self-Correction Abilitycited by 45
  7. 2025Exploring the Impact of Temperature on Large Language Models:Hot or Cold?cited by 40
  8. 2024Self-Reflection in LLM Agents: Effects on Problem-Solving Performancecited by 38
  9. 2025SoftCoT++: Test-Time Scaling with Soft Chain-of-Thought Reasoningcited by 38
  10. 2025Optimizing Temperature for Language Models with Multi-Sample Inferencecited by 33
  11. 2024Easy Problems That LLMs Get Wrongcited by 29
  12. 2024PET-SQL: A Prompt-Enhanced Two-Round Refinement of Text-to-SQL with Cross-consistencycited by 26
38 more
  1. 2025Exploring the Roles of Large Language Models in Reshaping Transportation Systems: A Survey, Framework, and Roadmapcited by 26
  2. 2025LLM Probability Concentration: How Alignment Shrinks the Generative Horizoncited by 26
  3. 2025Thinking Slow, Fast: Scaling Inference Compute with Distilled Reasonerscited by 22
  4. 2025M1: Towards Scalable Test-Time Compute with Mamba Reasoning Modelscited by 21
  5. 2025LEMMA: Learning from Errors for MatheMatical Advancement in LLMscited by 20
  6. 2025LLMs Provide Unstable Answers to Legal Questionscited by 19
  7. 2025Self-Correction Bench: Uncovering and Addressing the Self-Correction Blind Spot in Large Language Modelscited by 19
  8. 2025Control the Temperature: Selective Sampling for Diverse and High-Quality LLM Outputscited by 15
  9. 2025StableToolBench-MirrorAPI: Modeling Tool Environments as Mirrors of 7,000+ Real-World APIscited by 14
  10. 2025G2: Guided Generation for Enhanced Output Diversity in LLMscited by 14
  11. 2024Cognitive Overload Attack:Prompt Injection for Long Contextcited by 13
  12. 2025Continuous Self-Improvement of Large Language Models by Test-time Training with Verifier-Driven Sample Selectioncited by 13
  13. 2024Dialectical Alignment: Resolving the Tension of 3H and Security Threats of LLMscited by 11
  14. 2025Enhance-A-Video: Better Generated Video for Freecited by 11
  15. 2025Revisiting Self-Consistency from Dynamic Distributional Alignment Perspective on Answer Aggregationcited by 11
  16. 2024Can Machines Think Like Humans? A Behavioral Evaluation of LLM Agents in Dictator Gamescited by 9
  17. 2024Automatic Generation of Question Hints for Mathematics Problems using Large Language Models in Educational Technologycited by 9
  18. 2025How Diversely Can Language Models Solve Problems? Exploring the Algorithmic Diversity of Model-Generated Codecited by 9
  19. 2025When Life Gives You Samples: The Benefits of Scaling up Inference Compute for Multilingual LLMscited by 9
  20. 2025Semantic-guided Diverse Decoding for Large Language Modelcited by 9
  21. 2025Let it Calm: Exploratory Annealed Decoding for Verifiable Reinforcement Learningcited by 9
  22. 2024FLEXTAF: Enhancing Table Reasoning with Flexible Tabular Formatscited by 8
  23. 2024Quantifying perturbation impacts for large language modelscited by 8
  24. 2025GPT-4.1 Sets the Standard in Automated Experiment Design Using Novel Python Librariescited by 8
  25. 2025Simulating Viva Voce Examinations to Evaluate Clinical Reasoning in Large Language Modelscited by 8
  26. 2026Learn-to-Distance: Distance Learning for Detecting LLM-Generated Textcited by 8
  27. 2026ToolFailBench: Diagnosing Tool-Use Failures in LLM Agentscited by 8
  28. 2025Statistical Hypothesis Testing for Auditing Robustness in Language Modelscited by 7
  29. 2025Don't throw the baby out with the bathwater: How and why deep learning for ARCcited by 7
  30. 2025What Does It Take to Be a Good AI Research Agent? Studying the Role of Ideation Diversitycited by 7
  31. 2026Hidden Measurement Error in LLM Pipelines Distorts Annotation, Evaluation, and Benchmarkingcited by 7
  32. 2024Beyond Labels: Aligning Large Language Models with Human-like Reasoningcited by 6
  33. 2024Rethinking the Uncertainty: A Critical Review and Analysis in the Era of Large Language Modelscited by 6
  34. 2025Has the Creativity of Large-Language Models peaked? An analysis of inter- and intra-LLM variabilitycited by 6
  35. 2025Adaptive Domain Modeling with Language Models: A Multi-Agent Approach to Task Planningcited by 6
  36. 2025ReCode: Improving LLM-based Code Repair with Fine-Grained Retrieval-Augmented Generationcited by 6
  37. 2024Leveraging Prior Experience: An Expandable Auxiliary Knowledge Base for Text-to-SQLcited by 5
  38. 2025Evaluating Binary Decision Biases in Large Language Models: Implications for Fair Agent-Based Financial Simulationscited by 5

50 most cited of 168 All 168 (JSON)