mlsanityData to 14 Oct 2026

Decomposing the Entropy-Performance Exchange: The Missing Keys to Unlocking Effective Reinforcement Learning

arXiv 2508.02260 ↗First posted 4 Aug 2025cs.CL cs.AIcited by 21

On the map: LLM reasoning → Reinforcement learning for reasoning

also in 1 more
Abstract (arXiv)

Recently, reinforcement learning with verifiable rewards (RLVR) has been widely used for enhancing the reasoning abilities of large language models (LLMs). A core challenge in RLVR involves managing the exchange between entropy and performance of policies. Despite the importance of this exchange, a fine-grained understanding of when and how this exchange operates most effectively remains limited. To bridge this gap, we conduct a systematic empirical analysis of the entropy-performance exchange mechanism of RLVR across different levels of granularity. Specifically, we first divide the training process into two distinct stages based on entropy dynamics, i.e., rising stage and plateau stage, and then systematically investigate how this mechanism varies across stage-level, instance-level, and token-level granularitiess. Our analysis reveals that, in the rising stage, entropy reduction in negative samples facilitates the learning of effective reasoning patterns, which in turn drives rapid performance gains. Moreover, in the plateau stage, learning efficiency strongly correlates with high-entropy tokens present in low-perplexity samples and those located at the end of sequences. Motivated by these findings, we propose two methods that dynamically adjust the reward signal using perplexity and positional information to focus RL updates on tokens that exhibit high learning potential, achieving improvements compared to the baseline methods on various LLMs.

Key facts. Cited by 21 papers in the map (data through 14 Oct 2026). Placed in Reinforcement learning for reasoning, one of 17 directions in LLM reasoning. Builds on 29 of its 34 references that are in the map. First posted on arXiv 4 Aug 2025.

Builds on

29 of 34 references.

Closest ancestors · 12

Same direction, newest first

  1. 2025Reinforcement Learning with Verifiable Rewards Implicitly Incentivizes Correct Reasoning in Base LLMs17 Jun
  2. 2025Beyond the 80/20 Rule: High-Entropy Minority Tokens Drive Effective Reinforcement Learning for LLM Reasoning2 Jun
  3. 2025The Entropy Mechanism of Reinforcement Learning for Reasoning Language Models28 May
  4. 2025Accelerating RL for LLM Reasoning with Optimal Advantage Regression27 May
  5. 2025Enhancing Efficiency and Exploration in Reinforcement Learning for LLMs24 May
  6. 2025The Unreasonable Effectiveness of Entropy Minimization in LLM Reasoning21 May
  7. 2025SEED-GRPO: Semantic Entropy Enhanced GRPO for Uncertainty-Aware Policy Optimization18 May
  8. 2025Reinforcement Learning for Reasoning in Large Language Models with One Training Example29 Apr
  9. 2025Does Reinforcement Learning Really Incentivize Reasoning Capacity in LLMs Beyond the Base Model?18 Apr
  10. 2025SimpleRL-Zoo: Investigating and Taming Zero Reinforcement Learning for Open Base Models in the Wild24 Mar
  11. 2025DAPO: An Open-Source LLM Reinforcement Learning System at Scale18 Mar
  12. 2017Proximal Policy Optimization Algorithms20 Jul

Other references · 17

  1. 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 9,379
  2. 2021Evaluating Large Language Models Trained on Codecited by 8,320
  3. 2021Measuring Mathematical Problem Solving With the MATH Datasetcited by 5,371
  4. 2018Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actorcited by 4,885
  5. 2023GPQA: A Graduate-Level Google-Proof Q&A Benchmarkcited by 2,879
  6. 2024HybridFlow: A Flexible and Efficient RLHF Frameworkcited by 2,057
  7. 2022Solving Quantitative Reasoning Problems with Language Modelscited by 1,392
  8. 2024Qwen2.5-Math Technical Report: Toward Mathematical Expert Model via Self-Improvementcited by 1,090
  9. 2017Reinforcement Learning with Deep Energy-Based Policiescited by 669
  10. 2025Skywork Open Reasoner 1 Technical Reportcited by 165
  11. 2025An Empirical Study on Eliciting and Improving R1-like Reasoning Modelscited by 78
  12. 2025Trust, But Verify: A Self-Verification Approach to Reinforcement Learning with Verifiable Rewardscited by 38
5 more
  1. 2024Forking Paths in Neural Text Generationcited by 34
  2. 2025Entropy-guided sequence weighting for efficient exploration in RL-based LLM fine-tuningcited by 13
  3. 2024Not Everything is All You Need: Toward Low-Redundant Optimization for Large Language Model Alignmentcited by 7
  4. 2024Entropy-Reinforced Planning with Large Language Models for Drug Discoverycited by 6
  5. 2025Spectral Bellman Method: Unifying Representation and Exploration in RLcited by 4

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 10; 2026: 11.

21 papers.

Sort:

By area:

  1. 2025Rethinking Entropy Interventions in RLVR: An Entropy Change Perspectivecited by 52
  2. 2025No Prompt Left Behind: Exploiting Zero-Variance Prompts in LLM Reinforcement Learning via Entropy-Guided Advantage Shapingcited by 50
  3. 2025The Choice of Divergence: A Neglected Key to Mitigating Diversity Collapse in Reinforcement Learning with Verifiable Rewardcited by 37
  4. 2025Revisiting Entropy in Reinforcement Learning for Large Reasoning Modelscited by 21
  5. 2025From Trial-and-Error to Improvement: A Systematic Analysis of LLM Exploration Mechanisms in RLVRcited by 17
  6. 2025From Code Foundation Models to Agents and Applications: A Comprehensive Survey and Practical Guide to Code Intelligencecited by 15
  7. 2026On the Direction of RLVR Updates for LLM Reasoning: Identification and Exploitationcited by 13
  8. 2025Clip-Low Increases Entropy and Clip-High Decreases Entropy in Reinforcement Learning of Large Language Modelscited by 12
  9. 2025Exploring Multi-Temperature Strategies for Token- and Rollout-Level Control in RLVRcited by 8
  10. 2026Unveiling Implicit Advantage Symmetry: Why GRPO Struggles with Exploration and Difficulty Adaptationcited by 7
  11. 2026Advantage Collapse in Group Relative Policy Optimization: Diagnosis and Mitigationcited by 7
  12. 2025ReLaX: Reasoning with Latent Exploration for Large Reasoning Modelscited by 5
9 more
  1. 2026Low-rank Optimization Trajectories Modeling for LLM RLVR Accelerationcited by 4
  2. 2026Nudging Beyond the Comfort Zone: Efficient Strategy-Guided Exploration for RLVRcited by 4
  3. 2026Understanding and Preventing Entropy Collapse in RLVR with On-Policy Entropy Flow Optimizationcited by 3
  4. 2026SAGE: Shaping Anchors for Guided Exploration in RLVR of LLMscited by 3
  5. 2025EARL: Entropy-Aware RL Alignment of LLMs for Reliable RTL Code Generationcited by 2
  6. 2026LAD: Learning Advantage Distribution for Reasoningcited by 2
  7. 2026Policy Improvement Reinforcement Learningcited by 2
  8. 2026Beyond Entropy: Correctness-Aware Advantage Shaping via Contrastive Policy Optimizationcited by 1
  9. 2026What Does Chain-of-Thought Entropy Measure? A Channel Audit of Scaffolding, Routing, and Contentcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Exploration vs Exploitation: Rethinking RLVR through Clipping, Entropy, and Spurious Reward
  2. 2025Low-probability Tokens Sustain Exploration in Reinforcement Learning with Verifiable Reward
  3. 2025Beyond High-Entropy Exploration: Correctness-Aware Low-Entropy Segment-Based Advantage Shaping for Reasoning LLMs
  4. 2026Asymmetric Advantage Modulation Calibrates Entropy Dynamics in RLVR
  5. 2026Addressing Performance Saturation for LLM RL via Precise Entropy Curve Control
  6. 2026Sparse but Critical: A Token-Level Analysis of Distributional Shifts in RLVR Fine-Tuning of LLMs
  7. 2025Reasoning with Exploration: An Entropy Perspective
  8. 2025Revisiting Entropy Regularization: Adaptive Coefficient Unlocks Its Potential for LLM Reinforcement Learning