# The Effect of Sampling Temperature on Problem Solving in Large Language Models

arXiv: https://arxiv.org/abs/2402.05201 · HTML: https://mlsanity.com/paper/2402.05201/

## Key facts

- Cited by 170 papers in the map (data through 14 Oct 2026).
- Builds on 19 of its 53 references that are in the map.
- First posted on arXiv 7 Feb 2024.

## Builds on (most cited first, 19 of 19)

- [GPT-4 Technical Report](https://mlsanity.com/paper/2303.08774/) (2023)
- [Llama 2: Open Foundation and Fine-Tuned Chat Models](https://mlsanity.com/paper/2307.09288/) (2023)
- [Chain-of-Thought Prompting Elicits Reasoning in Large Language Models](https://mlsanity.com/paper/2201.11903/) (2022)
- [Distilling the Knowledge in a Neural Network](https://mlsanity.com/paper/1503.02531/) (2015)
- [Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks](https://mlsanity.com/paper/1908.10084/) (2019)
- [Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challenge](https://mlsanity.com/paper/1803.05457/) (2018)
- [Large Language Models are Zero-Shot Reasoners](https://mlsanity.com/paper/2205.11916/) (2022)
- [HellaSwag: Can a Machine Really Finish Your Sentence?](https://mlsanity.com/paper/1905.07830/) (2019)
- [AGIEval: A Human-Centric Benchmark for Evaluating Foundation Models](https://mlsanity.com/paper/2304.06364/) (2023)
- [A Prompt Pattern Catalog to Enhance Prompt Engineering with ChatGPT](https://mlsanity.com/paper/2302.11382/) (2023)
- [MedMCQA : A Large-scale Multi-Subject Multi-Choice Dataset for Medical domain Question Answering](https://mlsanity.com/paper/2203.14371/) (2022)
- [Augmented Language Models: a Survey](https://mlsanity.com/paper/2302.07842/) (2023)
- [A Systematic Evaluation of Large Language Models of Code](https://mlsanity.com/paper/2202.13169/) (2022)
- [Recitation-Augmented Language Models](https://mlsanity.com/paper/2210.01296/) (2022)
- [From LSAT: The Progress and Challenges of Complex Reasoning](https://mlsanity.com/paper/2108.00648/) (2021)
- [Cost-Effective Hyperparameter Optimization for Large Language Model Generation Inference](https://mlsanity.com/paper/2303.04673/) (2023)
- [Retrieving Supporting Evidence for Generative Question Answering](https://mlsanity.com/paper/2309.11392/) (2023)
- [Contextual Temperature for Language Modeling](https://mlsanity.com/paper/2012.13575/) (2020)
- [Performance of ChatGPT on the US Fundamentals of Engineering Exam: Comprehensive Assessment of Proficiency and Potential Implications for Professional Environmental Engineering Practice](https://mlsanity.com/paper/2304.12198/) (2023)

## Built on by (most cited first, 30 of 170)

- [SimpleVLA-RL: Scaling VLA Training via Reinforcement Learning](https://mlsanity.com/paper/2509.09674/) (2025)
- [Trae Agent: An LLM-based Agent for Software Engineering with Test-time Scaling](https://mlsanity.com/paper/2507.23370/) (2025)
- [A Sober Look at Progress in Language Model Reasoning: Pitfalls and Paths to Reproducibility](https://mlsanity.com/paper/2504.07086/) (2025)
- [A Survey of Text-to-SQL in the Era of LLMs: Where are we, and where are we going?](https://mlsanity.com/paper/2408.05109/) (2024)
- [Towards Robust Agentic CUDA Kernel Benchmarking, Verification, and Optimization](https://mlsanity.com/paper/2509.14279/) (2025)
- [Large Language Models have Intrinsic Self-Correction Ability](https://mlsanity.com/paper/2406.15673/) (2024)
- [SoftCoT++: Test-Time Scaling with Soft Chain-of-Thought Reasoning](https://mlsanity.com/paper/2505.11484/) (2025)
- [Exploring the Impact of Temperature on Large Language Models:Hot or Cold?](https://mlsanity.com/paper/2506.07295/) (2025)
- [Self-Reflection in LLM Agents: Effects on Problem-Solving Performance](https://mlsanity.com/paper/2405.06682/) (2024)
- [Optimizing Temperature for Language Models with Multi-Sample Inference](https://mlsanity.com/paper/2502.05234/) (2025)
- [Easy Problems That LLMs Get Wrong](https://mlsanity.com/paper/2405.19616/) (2024)
- [PET-SQL: A Prompt-Enhanced Two-Round Refinement of Text-to-SQL with Cross-consistency](https://mlsanity.com/paper/2403.09732/) (2024)
- [Exploring the Roles of Large Language Models in Reshaping Transportation Systems: A Survey, Framework, and Roadmap](https://mlsanity.com/paper/2503.21411/) (2025)
- [LLM Probability Concentration: How Alignment Shrinks the Generative Horizon](https://mlsanity.com/paper/2506.17871/) (2025)
- [Thinking Slow, Fast: Scaling Inference Compute with Distilled Reasoners](https://mlsanity.com/paper/2502.20339/) (2025)
- [LEMMA: Learning from Errors for MatheMatical Advancement in LLMs](https://mlsanity.com/paper/2503.17439/) (2025)
- [M1: Towards Scalable Test-Time Compute with Mamba Reasoning Models](https://mlsanity.com/paper/2504.10449/) (2025)
- [Self-Correction Bench: Uncovering and Addressing the Self-Correction Blind Spot in Large Language Models](https://mlsanity.com/paper/2507.02778/) (2025)
- [LLMs Provide Unstable Answers to Legal Questions](https://mlsanity.com/paper/2502.05196/) (2025)
- [Control the Temperature: Selective Sampling for Diverse and High-Quality LLM Outputs](https://mlsanity.com/paper/2510.01218/) (2025)
- [StableToolBench-MirrorAPI: Modeling Tool Environments as Mirrors of 7,000+ Real-World APIs](https://mlsanity.com/paper/2503.20527/) (2025)
- [G2: Guided Generation for Enhanced Output Diversity in LLMs](https://mlsanity.com/paper/2511.00432/) (2025)
- [Cognitive Overload Attack:Prompt Injection for Long Context](https://mlsanity.com/paper/2410.11272/) (2024)
- [Continuous Self-Improvement of Large Language Models by Test-time Training with Verifier-Driven Sample Selection](https://mlsanity.com/paper/2505.19475/) (2025)
- [Dialectical Alignment: Resolving the Tension of 3H and Security Threats of LLMs](https://mlsanity.com/paper/2404.00486/) (2024)
- [Enhance-A-Video: Better Generated Video for Free](https://mlsanity.com/paper/2502.07508/) (2025)
- [Revisiting Self-Consistency from Dynamic Distributional Alignment Perspective on Answer Aggregation](https://mlsanity.com/paper/2502.19830/) (2025)
- [How Diversely Can Language Models Solve Problems? Exploring the Algorithmic Diversity of Model-Generated Code](https://mlsanity.com/paper/2503.00691/) (2025)
- [ToolFailBench: Diagnosing Tool-Use Failures in LLM Agents](https://mlsanity.com/paper/2607.04686/) (2026)
- [Let it Calm: Exploratory Annealed Decoding for Verifiable Reinforcement Learning](https://mlsanity.com/paper/2510.05251/) (2025)

## Abstract (arXiv)

In this research study, we empirically investigate the effect of sampling temperature on the performance of Large Language Models (LLMs) on various problem-solving tasks. We created a multiple-choice question-and-answer (MCQA) exam by randomly sampling problems from standard LLM benchmarks. Then, we used nine popular LLMs with five prompt-engineering techniques to solve the MCQA problems while increasing the sampling temperature from 0.0 to 1.6. Despite anecdotal reports to the contrary, our empirical results indicate that changes in temperature from 0.0 to 1.0 do not have a statistically significant impact on LLM performance for problem-solving tasks. In addition, these results appear to generalize across LLMs, prompt-engineering techniques, and problem domains. All code, data, and supplemental materials are available on GitHub at: https://github.com/matthewrenze/jhu-llm-temperature