mlsanityData to 14 Oct 2026

Let it Calm: Exploratory Annealed Decoding for Verifiable Reinforcement Learning

arXiv 2510.05251 ↗First posted 6 Oct 2025cs.CL cs.LGcited by 10

On the map: LLM reasoning → Reinforcement learning for reasoning

also in 1 more
Abstract (arXiv)

Reinforcement learning with verifiable rewards (RLVR) is a powerful paradigm for enhancing the reasoning capabilities of large language models (LLMs), yet its success hinges on effective exploration. An ideal exploration strategy must navigate two fundamental challenges: it must preserve sample quality while also ensuring training stability. While standard fixed-temperature sampling is simple, it struggles to balance these competing demands, as high temperatures degrade sample quality and low temperatures limit discovery. In this work, we propose a simpler and more effective strategy, Exploratory Annealed Decoding (EAD), grounded in the insight that exploration is most impactful on early tokens which define a sequence's semantic direction. EAD implements an intuitive **explore-at-the-beginning, exploit-at-the-end** strategy by annealing the sampling temperature from high to low during generation. This dynamic schedule encourages meaningful, high-level diversity at the start, then gradually lowers the temperature to preserve sample quality and keep the sampling distribution close to the target policy, which is essential for stable training. We demonstrate that EAD is a lightweight, plug-and-play method that significantly improves sample efficiency, consistently outperforming fixed-temperature sampling across various RLVR algorithms and model sizes. Our work suggests that aligning exploration with the natural dynamics of sequential generation offers a robust path to improving LLM reasoning.

Key facts. Cited by 10 papers in the map (data through 14 Oct 2026). Placed in Reinforcement learning for reasoning, one of 17 directions in LLM reasoning. Builds on 60 of its 71 references that are in the map. First posted on arXiv 6 Oct 2025.

Builds on

60 of 71 references.

Closest ancestors · 29

Same direction, newest first

  1. 2025TreePO: Bridging the Gap of Policy Optimization and Efficacy and Inference Efficiency with Heuristic Tree-based Modeling24 Aug
  2. 2025Pass@k Training for Adaptively Balancing Exploration and Exploitation of Large Reasoning Models14 Aug
  3. 2025Sample More to Think Less: Group Filtered Policy Optimization for Concise Reasoning13 Aug
  4. 2025From Trial-and-Error to Improvement: A Systematic Analysis of LLM Exploration Mechanisms in RLVR11 Aug
  5. 2025OctoThinker: Mid-training Incentivizes Reinforcement Learning Scaling25 Jun
  6. 2025Reasoning with Exploration: An Entropy Perspective17 Jun
  7. 2025Spurious Rewards: Rethinking Training Signals in RLVR12 Jun
  8. 2025Act Only When It Pays: Efficient Reinforcement Learning for LLM Reasoning via Selective Rollouts2 Jun
  9. 2025Beyond the 80/20 Rule: High-Entropy Minority Tokens Drive Effective Reinforcement Learning for LLM Reasoning2 Jun
  10. 2025Maximizing Confidence Alone Improves Reasoning28 May
  11. 2025The Entropy Mechanism of Reinforcement Learning for Reasoning Language Models28 May
  12. 2025Learning to Reason without External Rewards26 May
17 more
  1. 2025The Unreasonable Effectiveness of Entropy Minimization in LLM Reasoning21 May
  2. 2025Reinforcement Learning for Reasoning in Large Language Models with One Training Example29 Apr
  3. 2025TTRL: Test-Time Reinforcement Learning22 Apr
  4. 2025Improving RL Exploration for LLM Reasoning through Retrospective Replay19 Apr
  5. 2025Does Reinforcement Learning Really Incentivize Reasoning Capacity in LLMs Beyond the Base Model?18 Apr
  6. 2025Not All Rollouts are Useful: Down-Sampling Rollouts in LLM Reinforcement Learning18 Apr
  7. 2025A Minimalist Approach to LLM Reasoning: from Rejection Sampling to Reinforce15 Apr
  8. 2025Echo Chamber: RL Post-training Amplifies Behaviors Learned in Pretraining10 Apr
  9. 2025VAPO: Efficient and Reliable Reinforcement Learning for Advanced Reasoning Tasks7 Apr
  10. 2025Open-Reasoner-Zero: An Open Source Approach to Scaling Up Reinforcement Learning on the Base Model31 Mar
  11. 2025Understanding R1-Zero-Like Training: A Critical Perspective26 Mar
  12. 2025SimpleRL-Zoo: Investigating and Taming Zero Reinforcement Learning for Open Base Models in the Wild24 Mar
  13. 2025DAPO: An Open-Source LLM Reinforcement Learning System at Scale18 Mar
  14. 2025DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning22 Jan
  15. 2025Kimi k1.5: Scaling Reinforcement Learning with LLMs22 Jan
  16. 2024Tulu 3: Pushing Frontiers in Open Language Model Post-Training22 Nov
  17. 2017Proximal Policy Optimization Algorithms20 Jul

Other references · 31

  1. 2024The Llama 3 Herd of Modelscited by 14,715
  2. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,459
  3. 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 9,379
  4. 2025Qwen3 Technical Reportcited by 8,820
  5. 2020Measuring Massive Multitask Language Understandingcited by 7,963
  6. 2022Self-Consistency Improves Chain of Thought Reasoning in Language Modelscited by 5,708
  7. 2018Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actorcited by 4,885
  8. 2019The Curious Case of Neural Text Degenerationcited by 2,916
  9. 2019Fine-Tuning Language Models from Human Preferencescited by 2,150
  10. 2024Qwen2 Technical Reportcited by 2,120
  11. 2024Scaling LLM Test-Time Compute Optimally can be More Effective than Scaling Model Parameterscited by 1,986
  12. 2022Scaling Laws for Reward Model Overoptimizationcited by 1,095
19 more
  1. 2022Discovering Language Model Behaviors with Model-Written Evaluationscited by 836
  2. 2025Group Sequence Policy Optimizationcited by 835
  3. 2025rStar-Math: Small LLMs Can Master Math Reasoning with Self-Evolved Deep Thinkingcited by 313
  4. 2025Deep Think with Confidencecited by 205
  5. 2024The Effect of Sampling Temperature on Problem Solving in Large Language Modelscited by 170
  6. 2022Exploration in Deep Reinforcement Learning: A Surveycited by 166
  7. 2025A Sober Look at Progress in Language Model Reasoning: Pitfalls and Paths to Reproducibilitycited by 93
  8. 2025T1: Advancing Language Model Reasoning through Reinforcement Learning and Inference Scalingcited by 77
  9. 2024Adaptive Inference-Time Compute: LLMs Can Predict if They Can Do Better, Even Mid-Generationcited by 64
  10. 2024EDT: Improving Large Language Models' Generation by Entropy-based Dynamic Temperature Samplingcited by 45
  11. 2025Beyond Reward Hacking: Causal Rewards for Large Language Model Alignmentcited by 42
  12. 2021Batch size-invariance for policy optimizationcited by 35
  13. 2025LLM Probability Concentration: How Alignment Shrinks the Generative Horizoncited by 26
  14. 2024Know When To Stop: A Study of Semantic Drift in Text Generationcited by 18
  15. 2025Lost at the Beginning of Reasoningcited by 15
  16. 2025Heterogeneous Adaptive Policy Optimization: Tailoring Optimization to Every Token's Naturecited by 11
  17. 2025Enabling Autoregressive Models to Fill In Masked Tokenscited by 7
  18. 2021Simulated annealing for optimization of graphs and sequencescited by 5
  19. 2025Is Exploration or Optimization the Problem for Deep Reinforcement Learning?cited by 1

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 10.

10 papers.

  1. 2026Look Inward to Explore Outward: Learning Temperature Policy from LLM Internal States via Hierarchical RLcited by 27
  2. 2026SAGE-OPD: Selective Agent-Guided Intervention for Multi-Turn On-Policy Distillationcited by 15
  3. 2026Synthetic Sandbox for Training Machine Learning Engineering Agentscited by 11
  4. 2026ERPO: Token-Level Entropy-Regulated Policy Optimization for Large Reasoning Modelscited by 6
  5. 2026EBPO: Empirical Bayes Shrinkage for Stabilizing Group-Relative Policy Optimizationcited by 5
  6. 2026Where Rollouts Begin: Low-Load, High-Leverage First-Token Diversification for RLVRcited by 2
  7. 2026TARo: Token-level Adaptive Routing for LLM Test-time Alignmentcited by 1
  8. 2026Smaller Models are Natural Explorers for Policy-Level Diversity in GRPOcited by 1
  9. 2026TGRL: Temperature-Grouped Reinforcement Learning for Efficient Exploration in LLMscited by 0
  10. 2026Sharpening Tax in Post-Trainingcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026Decoupling Exploration from Optimization in RLVR
  2. 2026Adaptive Decoding via Test-Time Policy Learning for Self-Improving Generation
  3. 2026Re2: Unlocking LLM Reasoning via Reinforcement Learning with Re-solving
  4. 2025Efficient Reinforcement Learning for Large Language Models with Intrinsic Exploration
  5. 2026From Base Rollouts to RL Reasoning: A Budgeted Search Perspective
  6. 2025Low-probability Tokens Sustain Exploration in Reinforcement Learning with Verifiable Reward
  7. 2025CDE: Curiosity-Driven Exploration for Efficient Reinforcement Learning in Large Language Models
  8. 2026How You Begin is How You Reason: Driving Exploration in RLVR via Prefix-Tuned Priors