mlsanityData to 14 Oct 2026

Long Horizon Temperature Scaling

arXiv 2302.03686 ↗First posted 7 Feb 2023cs.LG cs.AIcited by 18

Not on the map. Why?

Abstract (arXiv)

Temperature scaling is a popular technique for tuning the sharpness of a model distribution. It is used extensively for sampling likely generations and calibrating model uncertainty, and even features as a controllable parameter to many large language models in deployment. However, autoregressive models rely on myopic temperature scaling that greedily optimizes the next token. To address this, we propose Long Horizon Temperature Scaling (LHTS), a novel approach for sampling from temperature-scaled joint distributions. LHTS is compatible with all likelihood-based models, and optimizes for the long horizon likelihood of samples. We derive a temperature-dependent LHTS objective, and show that finetuning a model on a range of temperatures produces a single model capable of generation with a controllable long horizon temperature parameter. We experiment with LHTS on image diffusion models and character/language autoregressive models, demonstrating advantages over myopic temperature scaling in likelihood and sample quality, and showing improvements in accuracy on a multiple choice analogy task by 10%.

Key facts. Cited by 18 papers in the map (data through 14 Oct 2026). Builds on 18 of its 31 references that are in the map. First posted on arXiv 7 Feb 2023.

Builds on

18 of 31 references.

References · 18

  1. 2020Language Models are Few-Shot Learnerscited by 32,419
  2. 2020Denoising Diffusion Probabilistic Modelscited by 21,076
  3. 2017On Calibration of Modern Neural Networkscited by 4,960
  4. 2013Efficient Estimation of Word Representations in Vector Spacecited by 4,375
  5. 2021On the Opportunities and Risks of Foundation Modelscited by 4,033
  6. 2021Improved Denoising Diffusion Probabilistic Modelscited by 3,761
  7. 2019The Curious Case of Neural Text Degenerationcited by 2,916
  8. 2019BoolQ: Exploring the Surprising Difficulty of Natural Yes/No Questionscited by 2,183
  9. 2018Glow: Generative Flow with Invertible 1x1 Convolutionscited by 1,545
  10. 2018Don't Give Me the Details, Just the Summary! Topic-Aware Convolutional Neural Networks for Extreme Summarizationcited by 1,317
  11. 2018Hierarchical Neural Story Generationcited by 1,193
  12. 2020NVAE: A Deep Hierarchical Variational Autoencodercited by 637
6 more
  1. 2019Neural Text Generation with Unlikelihood Trainingcited by 503
  2. 2021MAUVE: Measuring the Gap Between Neural Text and Human Text using Divergence Frontierscited by 470
  3. 2019Measuring Calibration in Deep Learningcited by 446
  4. 2020Calibration of Pre-trained Transformerscited by 315
  5. 2022Quark: Controllable Text Generation with Reinforced Unlearningcited by 237
  6. 2016A Simple, Fast Diverse Decoding Algorithm for Neural Generationcited by 58

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 2; 2024: 7; 2025: 6; 2026: 3.

18 papers.

Sort:

By area:

  1. 2023Amortizing intractable inference in large language modelscited by 97
  2. 2024Probabilistic Inference in Language Models via Twisted Sequential Monte Carlocited by 90
  3. 2024Latent Space Chain-of-Embedding Enables Output-free LLM Self-Evaluationcited by 57
  4. 2025Diverse Preference Learning for Capabilities and Alignmentcited by 25
  5. 2025Fast Controlled Generation from Language Models with Adaptive Weighted Rejection Samplingcited by 21
  6. 2025Verifying Chain-of-Thought Reasoning via Its Computational Graphcited by 20
  7. 2024Controllable Generation via Locally Constrained Resamplingcited by 11
  8. 2024Post-Hoc Reversal: Are We Selecting Models Prematurely?cited by 8
  9. 2025Reflection-Window Decoding: Text Generation with Selective Refinementcited by 8
  10. 2024Calibrating Bayesian Learning via Regularization, Confidence Minimization, and Selective Inferencecited by 7
  11. 2025Enabling Autoregressive Models to Fill In Masked Tokenscited by 7
  12. 2025Temporal Score Rescaling for Temperature Sampling in Diffusion and Flow Modelscited by 7
6 more
  1. 2023The Robust Semantic Segmentation UNCV2023 Challenge Resultscited by 5
  2. 2024Improving Local Training in Federated Learning via Temperature Scalingcited by 5
  3. 2024GeoAI Reproducibility and Replicability: a computational and spatial perspectivecited by 3
  4. 2026CodeCircuit: Toward Inferring LLM-Generated Code Correctness via Attribution Graphscited by 3
  5. 2026The Well-Tempered Classifier: Some Elementary Properties of Temperature Scalingcited by 1
  6. 2026Diversify Diffusion with Temperature Sampling and Variance-Corrective Time Shiftingcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026Inference-Time Scaling of Diffusion Models via Progressive Seed Pruning
  2. 2025Exploring the Impact of Temperature on Large Language Models:Hot or Cold?
  3. 2025Inference-Time Scaling for Diffusion Models beyond Scaling Denoising Steps
  4. 2026Inference-Time Scaling for Visual AutoRegressive modeling by Searching Representative Samples
  5. 2025Inference-Time Scaling for Flow Models via Stochastic Generation and Rollover Budget Forcing
  6. 2025Optimizing Temperature for Language Models with Multi-Sample Inference
  7. 2024Bigger is not Always Better: Scaling Properties of Latent Diffusion Models
  8. 2025LSRS: Latent Scale Rejection Sampling for Visual Autoregressive Modeling