When Does the Best Sampling Temperature Rise with the Budget? Sufficient Conditions for Pass@k
Not on the map. Why?
Abstract (arXiv)
The temperature that maximizes pass@k is often low for a small sampling budget and higher for a large budget. This pattern has been reported from Codex through recent multi-sample inference studies. It is not an algebraic property of pass@k: as Slocum et al. (ICLR 2025) observe, for one fixed task the maximizing temperature is independent of k. Building on that fixed-task observation and the hard/easy-task explanation, we give a formal population-level sufficient condition for the aggregate pattern. For task X, let pt(X) be one-sample success probability at temperature t, and define the conditional log-success response mt(u)=mathbbE[dot pt(X)mid pt(X)=u]/u. If mt(u) is nonincreasing in current success probability, then the normalized temperature derivative of aggregate pass@k is nondecreasing in k. Consequently, derivative signs are nested across budgets; if each temperature-performance curve is strictly single-peaked, its unique maximizer is nondecreasing in k. The proof identifies the mechanism as a monotone-likelihood-ratio power tilt toward lower-success tasks. We derive a closed-form two-stratum phase diagram, including upward and downward regimes, and show that the marginal temperature derivative admits an exact Beta(2,k) kernel representation whose kernel concentrates at one-sample success of order 1/k. Interpreting that scale as task-level localization additionally requires a regular, nonvanishing density-response factor near zero. A signed-moment representation yields diagnostic shape restrictions, while a short appendix records exact discrete refinements of the existing multi-configuration allocation formulation. No language model is trained, and no model query is used as an experimental measurement: the contribution is a conditional theory of an established empirical phenomenon, with assumptions that can be tested in future work.
Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 10 of its 17 references that are in the map. First posted on arXiv 1 Aug 2026.
Builds on
10 of 17 references.
References · 10
- 2021Evaluating Large Language Models Trained on Codecited by 8,320
- 2024Inference-Aware Fine-Tuning for Best-of-N Sampling in Large Language Modelscited by 89
- 2025Pass@K Policy Optimization: Solving Harder Reinforcement Learning Problemscited by 87
- 2025How Do Large Language Monkeys Get Their Power (Laws)?cited by 43
- 2025Weight Ensembling Improves Reasoning in Language Modelscited by 38
- 2025Optimizing Temperature for Language Models with Multi-Sample Inferencecited by 33
- 2025Diverse Preference Learning for Capabilities and Alignmentcited by 25
- 2025On the Role of Temperature Sampling in Test-Time Scalingcited by 11
- 2026Why Pass@k Optimization Can Degrade Pass@1: Prompt Interference in LLM Post-trainingcited by 8
- 2026Learning Adaptive LLM Decodingcited by 3
Built on by
Not cited yet.