mlsanityData to 14 Oct 2026

Rethinking Entropy Interventions in RLVR: An Entropy Change Perspective

arXiv 2510.10150 ↗First posted 11 Oct 2025cs.LG cs.AIcited by 52

On the map: LLM reasoning → Reinforcement learning for reasoning

Abstract (arXiv)

Reinforcement Learning with Verifiable Rewards (RLVR) serves as a cornerstone technique for enhancing the reasoning capabilities of Large Language Models (LLMs). However, its training is often plagued by entropy collapse, a rapid decline in policy entropy that limits exploration and undermines training effectiveness. While recent works attempt to mitigate this issue via several heuristic entropy interventions, the underlying mechanisms remain poorly understood. In this work, we conduct comprehensive theoretical and empirical analyses of entropy dynamics in RLVR, offering two main insights: (1) We derive a tight analytical approximation for token-level entropy change at each update step, revealing four governing factors and providing a unified theoretical framework to explain how existing methods influence entropy; (2) We reveal a fundamental limitation of recent approaches: they rely on heuristic adjustments to one or two of these factors, leaving other relevant factors unconsidered, thus inherently limiting their effectiveness. Motivated by these findings, we propose STEER, a principled entropy-modulation method that adaptively reweights tokens based on theoretically-estimated entropy variations. Extensive experiments across six mathematical reasoning and three coding benchmarks demonstrate that STEER effectively mitigates entropy collapse and consistently outperforms state-of-the-art baselines.

Key facts. Cited by 52 papers in the map (data through 14 Oct 2026). Placed in Reinforcement learning for reasoning, one of 17 directions in LLM reasoning. Builds on 50 of its 67 references that are in the map. First posted on arXiv 11 Oct 2025.

Builds on

50 of 67 references.

Closest ancestors · 24

Same direction, newest first

  1. 2025A Survey of Reinforcement Learning for Large Reasoning Models10 Sep
  2. 2025Outcome-based Exploration for LLM Reasoning8 Sep
  3. 2025Emergent Hierarchical Reasoning in LLMs through Reinforcement Learning3 Sep
  4. 2025DCPO: Dynamic Clipping Policy Optimization2 Sep
  5. 2025CURE: Critical-Token-Guided Re-Concatenation for Entropy-Collapse Prevention14 Aug
  6. 2025Decomposing the Entropy-Performance Exchange: The Missing Keys to Unlocking Effective Reinforcement Learning4 Aug
  7. 2025EDGE-GRPO: Entropy-Driven GRPO with Guided Error Correction for Advantage Diversity29 Jul
  8. 2025Stabilizing Knowledge, Promoting Reasoning: Dual-Token Constraints for RLVR21 Jul
  9. 2025Reasoning with Exploration: An Entropy Perspective17 Jun
  10. 2025Rewarding the Unlikely: Lifting GRPO Beyond Distribution Sharpening3 Jun
  11. 2025Beyond the 80/20 Rule: High-Entropy Minority Tokens Drive Effective Reinforcement Learning for LLM Reasoning2 Jun
  12. 2025The Surprising Effectiveness of Negative Reinforcement in LLM Reasoning2 Jun
12 more
  1. 2025ProRL: Prolonged Reinforcement Learning Expands Reasoning Boundaries in Large Language Models30 May
  2. 2025On-Policy RL with Optimal Reward Baseline29 May
  3. 2025The Entropy Mechanism of Reinforcement Learning for Reasoning Language Models28 May
  4. 2025Does Reinforcement Learning Really Incentivize Reasoning Capacity in LLMs Beyond the Base Model?18 Apr
  5. 2025GPG: A Simple and Strong Reinforcement Learning Baseline for Model Reasoning3 Apr
  6. 2025Open-Reasoner-Zero: An Open Source Approach to Scaling Up Reinforcement Learning on the Base Model31 Mar
  7. 2025SimpleRL-Zoo: Investigating and Taming Zero Reinforcement Learning for Open Base Models in the Wild24 Mar
  8. 2025DAPO: An Open-Source LLM Reinforcement Learning System at Scale18 Mar
  9. 2025Process Reinforcement through Implicit Rewards3 Feb
  10. 2025Kimi k1.5: Scaling Reinforcement Learning with LLMs22 Jan
  11. 2024Tulu 3: Pushing Frontiers in Open Language Model Post-Training22 Nov
  12. 2017Proximal Policy Optimization Algorithms20 Jul

Other references · 26

  1. 2024The Llama 3 Herd of Modelscited by 14,715
  2. 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 9,379
  3. 2025Qwen3 Technical Reportcited by 8,820
  4. 2021Measuring Mathematical Problem Solving With the MATH Datasetcited by 5,371
  5. 2018Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actorcited by 4,885
  6. 2016Asynchronous Methods for Deep Reinforcement Learningcited by 2,258
  7. 2024HybridFlow: A Flexible and Efficient RLHF Frameworkcited by 2,057
  8. 2024LiveCodeBench: Holistic and Contamination Free Evaluation of Large Language Models for Codecited by 2,004
  9. 2022Solving Quantitative Reasoning Problems with Language Modelscited by 1,392
  10. 2024OlympiadBench: A Challenging Benchmark for Promoting AGI with Olympiad-Level Bilingual Multimodal Scientific Problemscited by 1,136
  11. 2025Demystifying Long Chain-of-Thought Reasoning in LLMscited by 326
  12. 2025Skywork Open Reasoner 1 Technical Reportcited by 165
14 more
  1. 2022CIRS: Bursting Filter Bubbles by Counterfactual Interactive Recommender Systemcited by 38
  2. 2023Alleviating Matthew Effect of Offline Reinforcement Learning in Interactive Recommendationcited by 38
  3. 2025Table-R1: Region-based Reinforcement Learning for Table Understandingcited by 37
  4. 2025Curriculum-RLAIF: Curriculum Alignment with Reinforcement Learning from AI Feedbackcited by 32
  5. 2025Semantic-Aware Logical Reasoning via a Semiotic Frameworkcited by 32
  6. 2024How Do Recommendation Models Amplify Popularity Bias? An Analysis from the Spectral Perspectivecited by 21
  7. 2025A Survey of Large Language Models for Text-Guided Molecular Discovery: from Molecule Generation to Optimizationcited by 13
  8. 2025POLO: Preference-Guided Multi-Turn Reinforcement Learning for Lead Optimizationcited by 11
  9. 2025LoRA-MGPO: Mitigating Double Descent in Low-Rank Adaptation via Momentum-Guided Perturbation Optimizationcited by 10
  10. 2025SPARK: Synergistic Policy And Reward Co-Evolving Frameworkcited by 10
  11. 2024PSL: Rethinking and Improving Softmax Loss from Pairwise Perspective for Recommendationcited by 9
  12. 2025Breaking the Top-K Barrier: Advancing Top-K Ranking Metrics Optimization in Recommender Systemscited by 9
  13. 2025When Maximum Entropy Misleads Policy Optimizationcited by 7
  14. 2025From Emergence to Control: Probing and Modulating Self-Reflection in Language Modelscited by 6

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 9; 2026: 43.

52 papers.

Sort:

By area:

  1. 2026Maximum Likelihood Reinforcement Learningcited by 36
  2. 2025Unlocking Exploration in RLVR: Uncertainty-aware Advantage Shaping for Deeper Reasoningcited by 34
  3. 2026Logical Phase Transitions: Understanding Collapse in LLM Logical Reasoningcited by 25
  4. 2026ExpSeek: Self-Triggered Experience Seeking for Web Agentscited by 23
  5. 2025Adaptation of Agentic AI: A Survey of Post-Training, Memory, and Skillscited by 22
  6. 2025SofT-GRPO: Surpassing Discrete-Token LLM Reinforcement Learning via Gumbel-Reparameterized Soft-Thinking Policy Optimizationcited by 21
  7. 2025Rethinking Sample Polarity in Reinforcement Learning with Verifiable Rewardscited by 19
  8. 2026Rebellious Student: Reversing Teacher Signals for Reasoning Exploration with Self-Distilled RLVRcited by 15
  9. 2025Scheduling Your LLM Reinforcement Learning with Reasoning Treescited by 12
  10. 2026F-GRPO: Don't Let Your Policy Learn the Obvious and Forget the Rarecited by 11
  11. 2026Reinforced Efficient Reasoning via Semantically Diverse Explorationcited by 8
  12. 2026LaDi-RL: Latent Diffusion Reasoning Prevents Entropy Collapse in Reinforcement Learningcited by 7
38 more
  1. 2026Unveiling Implicit Advantage Symmetry: Why GRPO Struggles with Exploration and Difficulty Adaptationcited by 7
  2. 2026Mid-Think: Training-Free Intermediate-Budget Reasoning via Token-Level Triggerscited by 6
  3. 2026One Refiner to Unlock Them All: Inference-Time Reasoning Elicitation via Reinforcement Query Refinementcited by 6
  4. 2025Beyond Reasoning Gains: Mitigating General-Capability Forgetting in Large Reasoning Modelscited by 5
  5. 2026Orchestrating Tokens and Sequences: Dynamic Hybrid Policy Optimization for RLVRcited by 5
  6. 2026TSPO: Breaking the Double Homogenization Dilemma in Multi-turn Search Policy Optimizationcited by 5
  7. 2026When Domains Interact: Asymmetric and Order-Sensitive Cross-Domain Effects in Reinforcement Learning for Reasoningcited by 5
  8. 2026Flexible Entropy Control in RLVR with a Gradient-Preserving Perspectivecited by 4
  9. 2026MolMem: Memory-Augmented Agentic Reinforcement Learning for Sample-Efficient Molecular Optimizationcited by 4
  10. 2026Meta-Cognitive Memory Policy Optimization for Long-Horizon LLM Agentscited by 4
  11. 2025Entropy-Guided Reasoning Compressioncited by 3
  12. 2026Targeted Exploration via Unified Entropy Control for Reinforcement Learningcited by 3
  13. 2026Understanding and Preventing Entropy Collapse in RLVR with On-Policy Entropy Flow Optimizationcited by 3
  14. 2026STARE: Surprisal-Guided Token-Level Advantage Reweighting for Policy Entropy Stabilitycited by 3
  15. 2026Which Tokens Matter? Adaptive Token Selection for RLVR with the Relative Surprisal Indexcited by 3
  16. 2026Matching Supervision to the Student's Learning Capacity: A Unified Framework for On-Policy Self-Distillationcited by 3
  17. 2026When Images Look Right and Retrieve Wrong: Coverage-Guided Cross-Scale Re-Indexing for Knowledge-Faithful Generative Perceptioncited by 3
  18. 2026Rubric-Guided Process Reward for Stepwise Model Routingcited by 2
  19. 2026BODHI: Do LLMs Branch Out and Discover Heterogeneous Inferences?cited by 2
  20. 2026All for 1-Bit: Towards Genuine 1-Bit Post-Training Quantization for LLMscited by 2
  21. 2025CoRPO: Adding a Correctness Bias to GRPO Improves Generalizationcited by 1
  22. 2026Token-Level Policy Optimization: Linking Group-Level Rewards to Token-Level Aggregation via Sequence-Level Likelihoodcited by 1
  23. 2026Clipping Bottleneck: Stabilizing RLVR via Stochastic Recovery of Near-Boundary Signalscited by 1
  24. 2026Smaller Models are Natural Explorers for Policy-Level Diversity in GRPOcited by 1
  25. 2026Beyond Entropy: Learning from Token-Level Distributional Deviations for LLM Reasoningcited by 1
  26. 2026Search for Truth from Reasoning: A Dynamic Representation Editing Framework for Steering LLM Trajectoriescited by 1
  27. 2026Beyond Entropy: Correctness-Aware Advantage Shaping via Contrastive Policy Optimizationcited by 1
  28. 2026Unlearnable, or Unmeasured? On the Reliability of Difficulty Labels in RLVRcited by 1
  29. 2025GAPO: Robust Advantage Estimation for Real-World Code LLMscited by 0
  30. 2026Taming Extreme Tokens: Covariance-Aware GRPO with Gaussian-Kernel Advantage Reweightingcited by 0
  31. 2026GeoFaith: A Spatio-Temporal Dual View of Faithful Chain-of-Thoughtcited by 0
  32. 2026The Pitfall of Scaling Up: Uncovering and Mitigating Popularity Bias Amplification in Scaling Transformer-based Recommenderscited by 0
  33. 2026BV-Blend: Uncertainty-Weighted Historical Baselines for Stable Critic-Free RL with Verifiable Rewardscited by 0
  34. 2026ACPO: Asymmetric Credit Policy Optimization via Mode-Local Entropy Surrogatecited by 0
  35. 2026Sampling via Decision-Flow: Training-Free Extraction of Improved Latent Reasoning Paths in Large Language Modelscited by 0
  36. 2026COEVO: Co-Evolving Context and Parameters for Recursive Self-Improvementcited by 0
  37. 2026Surprising Success, Repeated Failure: Entropy-Guided Credit Assignment for Exploration in LLM Reasoningcited by 0
  38. 2026E2-OPSD: Taming Entropy Overshoot in On-Policy Self-Distillationcited by 0

50 most cited of 52 All 52 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2025Revisiting Entropy in Reinforcement Learning for Large Reasoning Models
  2. 2025Revisiting Entropy Regularization: Adaptive Coefficient Unlocks Its Potential for LLM Reinforcement Learning
  3. 2026Asymmetric Advantage Modulation Calibrates Entropy Dynamics in RLVR
  4. 2026Addressing Performance Saturation for LLM RL via Precise Entropy Curve Control
  5. 2026A Comparative Theoretical Analysis of Entropy Control Methods in Reinforcement Learning
  6. 2025Exploration vs Exploitation: Rethinking RLVR through Clipping, Entropy, and Spurious Reward
  7. 2025Low-probability Tokens Sustain Exploration in Reinforcement Learning with Verifiable Reward
  8. 2025Beyond High-Entropy Exploration: Correctness-Aware Low-Entropy Segment-Based Advantage Shaping for Reasoning LLMs