# Exploring Multi-Temperature Strategies for Token- and Rollout-Level Control in RLVR

arXiv: https://arxiv.org/abs/2510.08892 · HTML: https://mlsanity.com/paper/2510.08892/

## Key facts

- Cited by 8 papers in the map (data through 14 Oct 2026).
- Placed in Reinforcement learning for reasoning, one of 17 directions in LLM reasoning.
- Builds on 18 of its 23 references that are in the map.
- First posted on arXiv 10 Oct 2025.

## Placed in

- LLM reasoning / [Reinforcement learning for reasoning](https://mlsanity.com/lr/reinforcement-learning-for-reasoning/)

## Builds on (most cited first, 18 of 18)

- [Proximal Policy Optimization Algorithms](https://mlsanity.com/paper/1707.06347/) (2017)
- [Measuring Mathematical Problem Solving With the MATH Dataset](https://mlsanity.com/paper/2103.03874/) (2021)
- [Reasoning with Exploration: An Entropy Perspective](https://mlsanity.com/paper/2506.14758/) (2025)
- [Pass@k Training for Adaptively Balancing Exploration and Exploitation of Large Reasoning Models](https://mlsanity.com/paper/2508.10751/) (2025)
- [Increasing Diversity While Maintaining Accuracy: Text Data Generation with Large Language Models and Human Interventions](https://mlsanity.com/paper/2306.04140/) (2023)
- [Not All Rollouts are Useful: Down-Sampling Rollouts in LLM Reinforcement Learning](https://mlsanity.com/paper/2504.13818/) (2025)
- [Outcome-based Exploration for LLM Reasoning](https://mlsanity.com/paper/2509.06941/) (2025)
- [Jointly Reinforcing Diversity and Quality in Language Model Generations](https://mlsanity.com/paper/2509.02534/) (2025)
- [EDT: Improving Large Language Models' Generation by Entropy-based Dynamic Temperature Sampling](https://mlsanity.com/paper/2403.14541/) (2024)
- [Diversity-Aware Policy Optimization for Large Language Model Reasoning](https://mlsanity.com/paper/2505.23433/) (2025)
- [Stabilizing Knowledge, Promoting Reasoning: Dual-Token Constraints for RLVR](https://mlsanity.com/paper/2507.15778/) (2025)
- [Optimizing Temperature for Language Models with Multi-Sample Inference](https://mlsanity.com/paper/2502.05234/) (2025)
- [Decomposing the Entropy-Performance Exchange: The Missing Keys to Unlocking Effective Reinforcement Learning](https://mlsanity.com/paper/2508.02260/) (2025)
- [How Diversely Can Language Models Solve Problems? Exploring the Algorithmic Diversity of Model-Generated Code](https://mlsanity.com/paper/2503.00691/) (2025)
- [Dynamic Stochastic Decoding Strategy for Open-Domain Dialogue Generation](https://mlsanity.com/paper/2406.07850/) (2024)
- [Adaptive Decoding via Latent Preference Optimization](https://mlsanity.com/paper/2411.09661/) (2024)
- [Improving Diversity in Language Models: When Temperature Fails, Change the Loss](https://mlsanity.com/paper/2508.09654/) (2025)
- [Is Functional Correctness Enough to Evaluate Code Language Models? Exploring Diversity of Generated Codes](https://mlsanity.com/paper/2408.14504/) (2024)

## Built on by (most cited first, 8 of 8)

- [Look Inward to Explore Outward: Learning Temperature Policy from LLM Internal States via Hierarchical RL](https://mlsanity.com/paper/2602.13035/) (2026)
- [Beyond Mode Collapse: Distribution Matching for Diverse Reasoning](https://mlsanity.com/paper/2605.19461/) (2026)
- [When Sharpening Becomes Collapse: Sampling Bias and Semantic Coupling in RL with Verifiable Rewards](https://mlsanity.com/paper/2601.15609/) (2026)
- [Where Rollouts Begin: Low-Load, High-Leverage First-Token Diversification for RLVR](https://mlsanity.com/paper/2605.28295/) (2026)
- [Smaller Models are Natural Explorers for Policy-Level Diversity in GRPO](https://mlsanity.com/paper/2605.30789/) (2026)
- [Parameter Exploration for RLVR via Variational Learning](https://mlsanity.com/paper/2608.09805/) (2026)
- [Know Thy Reasoner: Not All Language Models Explore Alike](https://mlsanity.com/paper/2604.10827/) (2026)
- [Cool the Sampler, Not the Learner: Sampling Temperature Moves the Staleness Cliff of Importance-Corrected GRPO](https://mlsanity.com/paper/2609.36953/) (2026)

## Abstract (arXiv)

Reinforcement Learning has demonstrated substantial improvements in the reasoning abilities of Large Language Models (LLMs), exhibiting significant applicability across various domains. Recent research has identified that tokens within LLMs play distinct roles during reasoning tasks, categorizing them into high-entropy reasoning tokens and low-entropy knowledge tokens. Prior approaches have typically focused on restricting updates to indirectly encourage exploration, yet they do not explicitly facilitate exploratory behavior during the token generation stage itself. In this work, we introduce a complementary approach that explicitly promotes exploration during sampling by applying distinct temperature settings for different token types. Specifically, our method employs higher temperatures for reasoning tokens to actively encourage exploration, while retaining lower temperatures for knowledge tokens to maintain factual correctness. Furthermore, we systematically investigate various multi-temperature scheduling strategies and their impacts within reinforcement learning contexts. Empirical evaluations on several reasoning benchmarks demonstrate that our approach significantly enhances the reasoning performance of LLMs. The code is available at https://github.com/zhmzm/Multi_Temperature_Verl.git.