# Contextual Temperature for Language Modeling

arXiv: https://arxiv.org/abs/2012.13575 · HTML: https://mlsanity.com/paper/2012.13575/

## Key facts

- Cited by 27 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 25 Dec 2020.

## Built on by (most cited first, 27 of 27)

- [The Effect of Sampling Temperature on Problem Solving in Large Language Models](https://mlsanity.com/paper/2402.05201/) (2024)
- [Non-Determinism of "Deterministic" LLM Settings](https://mlsanity.com/paper/2408.04667/) (2024)
- [Reasoning with Sampling: Your Base Model is Smarter Than You Think](https://mlsanity.com/paper/2510.14901/) (2025)
- [Large Language Models have Intrinsic Self-Correction Ability](https://mlsanity.com/paper/2406.15673/) (2024)
- [Scalable Power Sampling: Unlocking Efficient, Training-Free Reasoning for LLMs via Distribution Sharpening](https://mlsanity.com/paper/2601.21590/) (2026)
- [Adaptive Preference Scaling for Reinforcement Learning with Human Feedback](https://mlsanity.com/paper/2406.02764/) (2024)
- [Straight to the Gradient: Learning to Use Novel Tokens for Neural Text Generation](https://mlsanity.com/paper/2106.07207/) (2021)
- [Cognitive Overload Attack:Prompt Injection for Long Context](https://mlsanity.com/paper/2410.11272/) (2024)
- [Certified Self-Consistency: Statistical Guarantees and Test-Time Training for Reliable Reasoning in LLMs](https://mlsanity.com/paper/2510.17472/) (2025)
- [To Cool or not to Cool? Temperature Network Meets Large Foundation Models via DRO](https://mlsanity.com/paper/2404.04575/) (2024)
- [TreeSynth: Synthesizing Diverse Data from Scratch via Tree-Guided Subspace Partitioning](https://mlsanity.com/paper/2503.17195/) (2025)
- [Improving Local Training in Federated Learning via Temperature Scaling](https://mlsanity.com/paper/2401.09986/) (2024)
- [Sampling for Quality: Training-Free Reward-Guided LLM Decoding via Sequential Monte Carlo](https://mlsanity.com/paper/2604.16453/) (2026)
- [Can Large Language Models Invent Algorithms to Improve Themselves?: Algorithm Discovery for Recursive Self-Improvement through Reinforcement Learning](https://mlsanity.com/paper/2410.15639/) (2024)
- [Learning Natural Language Generation from Scratch](https://mlsanity.com/paper/2109.09371/) (2021)
- [A statistically consistent measure of semantic uncertainty using Language Models](https://mlsanity.com/paper/2502.00507/) (2025)
- [From Text to Pose to Image: Improving Diffusion Model Control and Quality](https://mlsanity.com/paper/2411.12872/) (2024)
- [MM-TS: Multi-Modal Temperature and Margin Schedules for Contrastive Learning with Long-Tail Data](https://mlsanity.com/paper/2603.08202/) (2026)
- [Fine-tune your Classifier: Finding Correlations With Temperature](https://mlsanity.com/paper/2210.09715/) (2022)
- [Synthesize, Partition, then Adapt: Eliciting Diverse Samples from Foundation Models](https://mlsanity.com/paper/2411.06722/) (2024)
- [Temperature in SLMs: Impact on Incident Categorization in On-Premises Environments](https://mlsanity.com/paper/2511.19464/) (2025)
- [Fusing Memory and Attention: A study on LSTM, Transformer and Hybrid Architectures for Symbolic Music Generation](https://mlsanity.com/paper/2603.21282/) (2026)
- [Thinking effort aligns between humans and reasoning models in abductive reasoning](https://mlsanity.com/paper/2609.01867/) (2026)
- [TeMo: Temperature Modulation for Multimodal Contrastive Learning](https://mlsanity.com/paper/2609.07540/) (2026)
- [Sampling via Decision-Flow: Training-Free Extraction of Improved Latent Reasoning Paths in Large Language Models](https://mlsanity.com/paper/2609.12317/) (2026)
- [Adaptive Power Sampling for LLM Reasoning](https://mlsanity.com/paper/2610.08563/) (2026)
- [Read the Room Before Your Model Generates](https://mlsanity.com/paper/2610.18820/) (2026)

## Abstract (arXiv)

Temperature scaling has been widely used as an effective approach to control the smoothness of a distribution, which helps the model performance in various tasks. Current practices to apply temperature scaling assume either a fixed, or a manually-crafted dynamically changing schedule. However, our studies indicate that the individual optimal trajectory for each class can change with the context. To this end, we propose contextual temperature, a generalized approach that learns an optimal temperature trajectory for each vocabulary over the context. Experimental results confirm that the proposed method significantly improves state-of-the-art language models, achieving a perplexity of 55.31 and 62.89 on the test set of Penn Treebank and WikiText-2, respectively. In-depth analyses show that the behaviour of the learned temperature schedules varies dramatically by vocabulary, and that the optimal schedules help in controlling the uncertainties. These evidences further justify the need for the proposed method and its advantages over fixed temperature schedules.