mlsanityData to 14 Oct 2026

Optimizing Temperature for Language Models with Multi-Sample Inference

arXiv 2502.05234 ↗First posted 7 Feb 2025cs.LG cs.AI cs.CLcited by 33

On the map: LLM reasoning → Test-time compute and efficient thinking

Abstract (arXiv)

Multi-sample aggregation strategies, such as majority voting and best-of-N sampling, are widely used in contemporary large language models (LLMs) to enhance predictive accuracy across various tasks. A key challenge in this process is temperature selection, which significantly impacts model performance. Existing approaches either rely on a fixed default temperature or require labeled validation data for tuning, which are often scarce and difficult to obtain. This paper addresses the challenge of automatically identifying the (near)-optimal temperature for different LLMs using multi-sample aggregation strategies, without relying on task-specific validation data. We provide a comprehensive analysis of temperature's role in performance optimization, considering variations in model architectures, datasets, task types, model sizes, and predictive accuracy. Furthermore, we propose a novel entropy-based metric for automated temperature optimization, which consistently outperforms fixed-temperature baselines. Additionally, we incorporate a stochastic process model to enhance interpretability, offering deeper insights into the relationship between temperature and model performance.

Key facts. Cited by 33 papers in the map (data through 14 Oct 2026). Placed in Test-time compute and efficient thinking, one of 17 directions in LLM reasoning. Builds on 35 of its 41 references that are in the map. First posted on arXiv 7 Feb 2025.

Builds on

35 of 41 references.

Closest ancestors · 2

Same direction, newest first

  1. 2024Scaling LLM Inference with Optimized Sample Compute Allocation29 Oct
  2. 2023Tree of Thoughts: Deliberate Problem Solving with Large Language Models17 May

Other references · 33

  1. 2024The Llama 3 Herd of Modelscited by 14,715
  2. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,459
  3. 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 9,379
  4. 2021Evaluating Large Language Models Trained on Codecited by 8,320
  5. 2023Efficient Memory Management for Large Language Model Serving with PagedAttentioncited by 5,944
  6. 2022Self-Consistency Improves Chain of Thought Reasoning in Language Modelscited by 5,708
  7. 2021Measuring Mathematical Problem Solving With the MATH Datasetcited by 5,371
  8. 2021Program Synthesis with Large Language Modelscited by 3,637
  9. 2023Mistral 7Bcited by 3,164
  10. 2019The Curious Case of Neural Text Degenerationcited by 2,916
  11. 2023A Survey on Large Language Model based Autonomous Agentscited by 2,242
  12. 2023Code Llama: Open Foundation Models for Codecited by 2,139
21 more
  1. 2024Qwen2.5-Coder Technical Reportcited by 1,203
  2. 2024DeepSeek-Coder: When the Large Language Model Meets Programming -- The Rise of Code Intelligencecited by 1,181
  3. 2023Math-Shepherd: Verify and Reinforce LLMs Step-by-step without Human Annotationscited by 928
  4. 2023Llemma: An Open Language Model For Mathematicscited by 388
  5. 2023Building Cooperative Embodied Agents Modularly with Large Language Modelscited by 237
  6. 2019Stochastic Beams and Where to Find Them: The Gumbel-Top-k Trick for Sampling Sequences Without Replacementcited by 211
  7. 2024OpenMathInstruct-2: Accelerating AI for Math with Massive Open-Source Instruction Datacited by 200
  8. 2024The Effect of Sampling Temperature on Problem Solving in Large Language Modelscited by 170
  9. 2024Turning Up the Heat: Min-p Sampling for Creative and Coherent LLM Outputscited by 150
  10. 2024From Decoding to Meta-Generation: Inference-time Algorithms for Large Language Modelscited by 137
  11. 2022Locally Typical Samplingcited by 111
  12. 2023Evaluating Open-Domain Question Answering in the Era of Large Language Modelscited by 111
  13. 2024Easy-to-Hard Generalization: Scalable Alignment Beyond Human Supervisioncited by 96
  14. 2024LLM and Simulation as Bilevel Optimizers: A New Paradigm to Advance Physical Scientific Discoverycited by 86
  15. 2024Planning In Natural Language Improves LLM Search For Code Generationcited by 83
  16. 2019Calibration of Encoder Decoder Models for Neural Machine Translationcited by 68
  17. 2024Lean-STaR: Learning to Interleave Thinking and Provingcited by 68
  18. 2024Calibrating Language Models with Adaptive Temperature Scalingcited by 58
  19. 2024EDT: Improving Large Language Models' Generation by Entropy-based Dynamic Temperature Samplingcited by 45
  20. 2024Dynamic Stochastic Decoding Strategy for Open-Domain Dialogue Generationcited by 10
  21. 2024Adaptive Decoding via Latent Preference Optimizationcited by 10

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 13; 2026: 20.

33 papers.

Sort:

By area:

  1. 2025Towards Reasoning Era: A Survey of Long Chain-of-Thought for Reasoning Large Language Modelscited by 384
  2. 2025BroRL: Scaling Reinforcement Learning via Broadened Explorationcited by 25
  3. 2025MedArabiQ: Benchmarking Large Language Models on Arabic Medical Taskscited by 15
  4. 2025Analyzable Chain-of-Musical-Thought Prompting for High-Fidelity Music Generationcited by 14
  5. 2025When Life Gives You Samples: The Benefits of Scaling up Inference Compute for Multilingual LLMscited by 9
  6. 2025Optimal Self-Consistency for Efficient Reasoning with Large Language Modelscited by 9
  7. 2025Estimating LLM Consistency: A User Baseline vs Surrogate Metricscited by 8
  8. 2025Exploring Multi-Temperature Strategies for Token- and Rollout-Level Control in RLVRcited by 8
  9. 2026Temperature as a Meta-Policy: Adaptive Temperature in LLM Reinforcement Learningcited by 8
  10. 2026Why Pass@k Optimization Can Degrade Pass@1: Prompt Interference in LLM Post-trainingcited by 8
  11. 2026The Consensus Trap: Rescuing Multi-Agent LLMs from Adversarial Majorities via Token-Level Collaborationcited by 8
  12. 2026Learning to Explore with Parameter-Space Noise: A Deep Dive into Parameter-Space Noise for Reinforcement Learning with Verifiable Rewardscited by 6
21 more
  1. 2025FinCoT: Grounding Chain-of-Thought in Expert Financial Reasoningcited by 5
  2. 2025Autonomous Radiotherapy Treatment Planning Using DOLA: A Privacy-Preserving, LLM-Based Optimization Agentcited by 4
  3. 2025Demystifying LLM-as-a-Judge: Analytically Tractable Model for Inference-Time Scalingcited by 4
  4. 2026Instinct vs. Reflection: Unifying Token and Verbalized Confidence in Multimodal Large Modelscited by 4
  5. 2026Learning Adaptive LLM Decodingcited by 3
  6. 2026The Necessity of Setting Temperature in LLM-as-a-Judgecited by 3
  7. 2026FLIPS: Instance-Fingerprinting for LLMs via Pseudo-random Sequencescited by 3
  8. 2025Large Language Models Develop Novel Social Biases Through Adaptive Explorationcited by 2
  9. 2026From Token to Token Pair: Efficient Prompt Compression for Large Language Models in Clinical Predictioncited by 2
  10. 2026AutoCRAT: Within-trajectory Joint Control of Stochasticity and Compute for LLM Reasoningcited by 2
  11. 2026UpSkill: Mutual Information Skill Learning for Structured Response Diversity in LLMscited by 1
  12. 2025TempPerturb-Eval: On the Joint Effects of Internal Temperature and External Perturbations in RAG Robustnesscited by 0
  13. 2026Temperature-Dependent Performance of Prompting Strategies in Extended Reasoning Large Language Modelscited by 0
  14. 2026Trajectory as the Teacher: Few-Step Discrete Flow Matching via Energy-Navigated Distillationcited by 0
  15. 2026Towards Trust Calibration in Socially Interactive Agents: Investigating Gendered Multimodal Behaviors Generation with LLMscited by 0
  16. 2026SonicSampler: Unified Tile-Aware Kernels for LLM Sampling and Speculative Verificationcited by 0
  17. 2026COMPAS: Difficulty-Aware Joint Search for Optimizing Code Generationcited by 0
  18. 2026Focus particles and scalar inferences across humans and language modelscited by 0
  19. 2026When Does the Best Sampling Temperature Rise with the Budget? Sufficient Conditions for Pass@kcited by 0
  20. 2026Composable Decoding on the Probability Simplex: Theory and Implementationcited by 0
  21. 2026Explore Broadly, Reason Sharply: Push Small Models toward the Frontier via Samplingcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Exploring the Impact of Temperature on Large Language Models:Hot or Cold?
  2. 2026A Guide to Large Language Models in Modeling and Simulation: From Core Techniques to Critical Challenges
  3. 2024A Probabilistic Perspective on Unlearning and Alignment for Large Language Models
  4. 2026Rethinking the Role of Temperature in Large Language Model Distillation
  5. 2025On the Role of Temperature Sampling in Test-Time Scaling
  6. 2025LEAD: Iterative Data Selection for Efficient LLM Instruction Tuning
  7. 2025Control the Temperature: Selective Sampling for Diverse and High-Quality LLM Outputs
  8. 2025TokenPowerBench: Benchmarking the Power Consumption of LLM Inference