mlsanityData to 14 Oct 2026

The Entropy Mechanism of Reinforcement Learning for Reasoning Language Models

arXiv 2505.22617 ↗First posted 28 May 2025cs.LG cs.AI cs.CLcited by 485

On the map: LLM reasoning → Reinforcement learning for reasoning

Abstract (arXiv)

This paper aims to overcome a major obstacle in scaling RL for reasoning with LLMs, namely the collapse of policy entropy. Such phenomenon is consistently observed across vast RL runs without entropy intervention, where the policy entropy dropped sharply at the early training stage, this diminished exploratory ability is always accompanied with the saturation of policy performance. In practice, we establish a transformation equation R=-a*e^H+b between entropy H and downstream performance R. This empirical law strongly indicates that, the policy performance is traded from policy entropy, thus bottlenecked by its exhaustion, and the ceiling is fully predictable H=0, R=-a+b. Our finding necessitates entropy management for continuous exploration toward scaling compute for RL. To this end, we investigate entropy dynamics both theoretically and empirically. Our derivation highlights that, the change in policy entropy is driven by the covariance between action probability and the change in logits, which is proportional to its advantage when using Policy Gradient-like algorithms. Empirical study shows that, the values of covariance term and entropy differences matched exactly, supporting the theoretical conclusion. Moreover, the covariance term stays mostly positive throughout training, further explaining why policy entropy would decrease monotonically. Through understanding the mechanism behind entropy dynamics, we motivate to control entropy by restricting the update of high-covariance tokens. Specifically, we propose two simple yet effective techniques, namely Clip-Cov and KL-Cov, which clip and apply KL penalty to tokens with high covariances respectively. Experiments show that these methods encourage exploration, thus helping policy escape entropy collapse and achieve better downstream performance.

Key facts. Cited by 485 papers in the map (data through 14 Oct 2026). Placed in Reinforcement learning for reasoning, one of 17 directions in LLM reasoning. Builds on 34 of its 57 references that are in the map. First posted on arXiv 28 May 2025.

Builds on

34 of 57 references.

Closest ancestors · 13

Same direction, newest first

  1. 2025The Unreasonable Effectiveness of Entropy Minimization in LLM Reasoning21 May
  2. 2025TTRL: Test-Time Reinforcement Learning22 Apr
  3. 2025Learning to Reason under Off-Policy Guidance21 Apr
  4. 2025Does Reinforcement Learning Really Incentivize Reasoning Capacity in LLMs Beyond the Base Model?18 Apr
  5. 2025Right Question is Already Half the Answer: Fully Unsupervised LLM Reasoning Incentivization8 Apr
  6. 2025Open-Reasoner-Zero: An Open Source Approach to Scaling Up Reinforcement Learning on the Base Model31 Mar
  7. 2025Understanding R1-Zero-Like Training: A Critical Perspective26 Mar
  8. 2025DAPO: An Open-Source LLM Reinforcement Learning System at Scale18 Mar
  9. 2025Process Reinforcement through Implicit Rewards3 Feb
  10. 2025Kimi k1.5: Scaling Reinforcement Learning with LLMs22 Jan
  11. 2024Free Process Rewards without Process Labels2 Dec
  12. 2024Tulu 3: Pushing Frontiers in Open Language Model Post-Training22 Nov
1 more
  1. 2017Proximal Policy Optimization Algorithms20 Jul

Other references · 21

  1. 2022Training language models to follow instructions with human feedbackcited by 15,971
  2. 2021Training Verifiers to Solve Math Word Problemscited by 10,109
  3. 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 9,379
  4. 2020Scaling Laws for Neural Language Modelscited by 6,586
  5. 2021Measuring Mathematical Problem Solving With the MATH Datasetcited by 5,371
  6. 2018Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actorcited by 4,885
  7. 2022Training Compute-Optimal Large Language Modelscited by 3,019
  8. 2015Trust Region Policy Optimizationcited by 2,580
  9. 2016Asynchronous Methods for Deep Reinforcement Learningcited by 2,258
  10. 2024HybridFlow: A Flexible and Efficient RLHF Frameworkcited by 2,057
  11. 2022Language Models (Mostly) Know What They Knowcited by 1,856
  12. 2022Solving Quantitative Reasoning Problems with Language Modelscited by 1,392
9 more
  1. 2024OlympiadBench: A Challenging Benchmark for Promoting AGI with Olympiad-Level Bilingual Multimodal Scientific Problemscited by 1,136
  2. 2022Scaling Laws for Reward Model Overoptimizationcited by 1,095
  3. 2017Reinforcement Learning with Deep Energy-Based Policiescited by 669
  4. 2024Omni-MATH: A Universal Olympiad Level Mathematic Benchmark For Large Language Modelscited by 284
  5. 2021Maximum Entropy RL (Provably) Solves Some Robust RL Problemscited by 176
  6. 2017Equivalence Between Policy Gradients and Soft Q-Learningcited by 173
  7. 2025ACECODER: Acing Coder RL via Automated Test-Case Synthesiscited by 93
  8. 2023Scaling laws for single-agent reinforcement learningcited by 41
  9. 2025Value-Based Deep RL Scales Predictablycited by 15

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 196; 2026: 289.

485 papers.

Sort:

By area:

  1. 2025InternVL3.5: Advancing Open-Source Multimodal Models in Versatility, Reasoning, and Efficiencycited by 1,510
  2. 2025Reasoning with Exploration: An Entropy Perspectivecited by 281
  3. 2025MiniMax-M1: Scaling Test-Time Compute Efficiently with Lightning Attentioncited by 263
  4. 2025Kimi Linear: An Expressive, Efficient Attention Architecturecited by 196
  5. 2025DiffuCoder: Understanding and Improving Masked Diffusion Models for Code Generationcited by 187
  6. 2025A Survey of Reinforcement Learning for Large Reasoning Modelscited by 180
  7. 2025SimpleVLA-RL: Scaling VLA Training via Reinforcement Learningcited by 151
  8. 2025SimpleTIR: End-to-End Reinforcement Learning for Multi-Turn Tool-Integrated Reasoningcited by 146
  9. 2025Pass@k Training for Adaptively Balancing Exploration and Exploitation of Large Reasoning Modelscited by 138
  10. 2025SDAR: A Synergistic Diffusion-AutoRegression Paradigm for Scalable Sequence Generationcited by 129
  11. 2025Geometric-Mean Policy Optimizationcited by 111
  12. 2026TIP: Token Importance in On-Policy Distillationcited by 104
38 more
  1. 2025The Art of Scaling Reinforcement Learning Compute for LLMscited by 101
  2. 2025The Invisible Leash: Why RLVR May or May Not Escape Its Origincited by 97
  3. 2025Critique-GRPO: Advancing LLM Reasoning with Natural Language and Numerical Feedbackcited by 90
  4. 2025RLPR: Extrapolating RLVR to General Domains without Verifierscited by 85
  5. 2026Scaling Reasoning Efficiently via Relaxed On-Policy Distillationcited by 82
  6. 2025Jointly Reinforcing Diversity and Quality in Language Model Generationscited by 80
  7. 2025Defeating the Training-Inference Mismatch via FP16cited by 66
  8. 2026CRISP: Compressed Reasoning via Iterative Self-Policy Distillationcited by 58
  9. 2025Beyond Pass@1: Self-Play with Variational Problem Synthesis Sustains RLVRcited by 54
  10. 2025rStar2-Agent: Agentic Reasoning Technical Reportcited by 54
  11. 2025Intern-S1: A Scientific Multimodal Foundation Modelcited by 53
  12. 2025Prosperity before Collapse: How Far Can Off-Policy RL Reach with Stale Data on LLMs?cited by 53
  13. 2025ExGRPO: Learning to Reason from Experiencecited by 53
  14. 2025Rethinking Entropy Interventions in RLVR: An Entropy Change Perspectivecited by 52
  15. 2025FLEX: Continuous Agent Evolution via Forward Learning from Experiencecited by 52
  16. 2025No Prompt Left Behind: Exploiting Zero-Variance Prompts in LLM Reinforcement Learning via Entropy-Guided Advantage Shapingcited by 50
  17. 2026Hierarchy-of-Groups Policy Optimization for Long-Horizon Agentic Taskscited by 49
  18. 2025BAPO: Stabilizing Off-Policy Reinforcement Learning for LLMs via Balanced Policy Optimization with Adaptive Clippingcited by 47
  19. 2025Evolving Language Models without Labels: Majority Drives Selection, Novelty Promotes Variationcited by 45
  20. 2025QuestA: Expanding Reasoning Capacity in LLMs via Question Augmentationcited by 44
  21. 2025Emergent Hierarchical Reasoning in LLMs through Reinforcement Learningcited by 42
  22. 2025From f(x) and g(x) to f(g(x)): LLMs Learn New Skills in RL by Composing Old Onescited by 42
  23. 2025MMedAgent-RL: Optimizing Multi-Agent Collaboration for Multimodal Medical Reasoningcited by 40
  24. 2025On Entropy Control in LLM-RL Algorithmscited by 40
  25. 2025FlowRL: Matching Reward Distributions for LLM Reasoningcited by 40
  26. 2026Embarrassingly Simple Self-Distillation Improves Code Generationcited by 40
  27. 2025Stabilizing Knowledge, Promoting Reasoning: Dual-Token Constraints for RLVRcited by 39
  28. 2025No Free Lunch: Rethinking Internal Feedback for LLM Reasoningcited by 37
  29. 2025The Choice of Divergence: A Neglected Key to Mitigating Diversity Collapse in Reinforcement Learning with Verifiable Rewardcited by 37
  30. 2026Rethinking the Trust Region in LLM Reinforcement Learningcited by 37
  31. 2025ETTRL: Balancing Exploration and Exploitation in LLM Test-Time Reinforcement Learning Via Entropy Mechanismcited by 36
  32. 2025Towards a Unified View of Large Language Model Post-Trainingcited by 36
  33. 2025RL-PLUS: Countering Capability Boundary Collapse of LLMs in Reinforcement Learning with Hybrid-policy Optimizationcited by 35
  34. 2025Reinforcement Learning Meets Large Language Models: A Survey of Advancements and Applications Across the LLM Lifecyclecited by 35
  35. 2025Unlocking Exploration in RLVR: Uncertainty-aware Advantage Shaping for Deeper Reasoningcited by 34
  36. 2025Time Is a Feature: Exploiting Temporal Dynamics in Diffusion Language Modelscited by 33
  37. 2025Demystifying Reinforcement Learning in Agentic Reasoningcited by 33
  38. 2026STEP3-VL-10B Technical Reportcited by 32

50 most cited of 485 All 485 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2025Beyond the 80/20 Rule: High-Entropy Minority Tokens Drive Effective Reinforcement Learning for LLM Reasoning
  2. 2026Understanding and Preventing Entropy Collapse in RLVR with On-Policy Entropy Flow Optimization
  3. 2025Revisiting Entropy Regularization: Adaptive Coefficient Unlocks Its Potential for LLM Reinforcement Learning
  4. 2026Rethinking RL for LLM Reasoning: It's Sparse Policy Selection, Not Capability Learning
  5. 2025SRFT: A Single-Stage Method with Supervised and Reinforcement Fine-Tuning for Reasoning
  6. 2026Internalize the Temperature: On-Policy Self-Distillation as Policy Reheater for Reinforcement Learning
  7. 2026ERPO: Token-Level Entropy-Regulated Policy Optimization for Large Reasoning Models
  8. 2026Entropy-Aware On-Policy Distillation of Language Models