mlsanityData to 14 Oct 2026

Clip-Low Increases Entropy and Clip-High Decreases Entropy in Reinforcement Learning of Large Language Models

arXiv 2509.26114 ↗First posted 30 Sep 2025cs.LGcited by 12

On the map: LLM reasoning → Reinforcement learning for reasoning

Abstract (arXiv)

Reinforcement learning with verifiable rewards (RLVR) has recently emerged as the leading approach for enhancing the reasoning capabilities of large language models (LLMs). However, RLVR is prone to entropy collapse, where the LLM quickly converges to a near-deterministic form, hindering exploration and progress during prolonged RL training. In this work, we reveal that the clipping mechanism in PPO and GRPO induces biases on entropy. Through theoretical and empirical analyses, we show that clip-low increases entropy, while clip-high decreases it. Further, under standard clipping parameters, the effect of clip-high dominates, resulting in an overall entropy reduction even when purely random rewards are provided to the RL algorithm. Our findings highlight an overlooked confounding factor in RLVR: independent of the reward signal, the clipping mechanism influences entropy, which in turn affects the reasoning behavior. Furthermore, our analysis demonstrates that clipping can be deliberately used to control entropy. Specifically, with a more aggressive clip-low value, one can increase entropy, promote exploration, and ultimately prevent entropy collapse in RLVR training.

Key facts. Cited by 12 papers in the map (data through 14 Oct 2026). Placed in Reinforcement learning for reasoning, one of 17 directions in LLM reasoning. Builds on 31 of its 38 references that are in the map. First posted on arXiv 30 Sep 2025.

Builds on

31 of 38 references.

Closest ancestors · 20

Same direction, newest first

  1. 2025Outcome-based Exploration for LLM Reasoning8 Sep
  2. 2025Pass@k Training for Adaptively Balancing Exploration and Exploitation of Large Reasoning Models14 Aug
  3. 2025Decomposing the Entropy-Performance Exchange: The Missing Keys to Unlocking Effective Reinforcement Learning4 Aug
  4. 2025Reasoning with Exploration: An Entropy Perspective17 Jun
  5. 2025Reinforcement Learning with Verifiable Rewards Implicitly Incentivizes Correct Reasoning in Base LLMs17 Jun
  6. 2025Spurious Rewards: Rethinking Training Signals in RLVR12 Jun
  7. 2025Rewarding the Unlikely: Lifting GRPO Beyond Distribution Sharpening3 Jun
  8. 2025The Surprising Effectiveness of Negative Reinforcement in LLM Reasoning2 Jun
  9. 2025ProRL: Prolonged Reinforcement Learning Expands Reasoning Boundaries in Large Language Models30 May
  10. 2025The Climb Carves Wisdom Deeper Than the Summit: On the Noisy Rewards in Learning to Reason28 May
  11. 2025The Entropy Mechanism of Reinforcement Learning for Reasoning Language Models28 May
  12. 2025Learning to Reason without External Rewards26 May
8 more
  1. 2025Navigate the Unknown: Enhancing LLM Reasoning with Intrinsic Motivation Guided Exploration23 May
  2. 2025The Unreasonable Effectiveness of Entropy Minimization in LLM Reasoning21 May
  3. 2025Reinforcement Learning for Reasoning in Large Language Models with One Training Example29 Apr
  4. 2025Does Reinforcement Learning Really Incentivize Reasoning Capacity in LLMs Beyond the Base Model?18 Apr
  5. 2025Understanding R1-Zero-Like Training: A Critical Perspective26 Mar
  6. 2025DAPO: An Open-Source LLM Reinforcement Learning System at Scale18 Mar
  7. 2025DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning22 Jan
  8. 2017Proximal Policy Optimization Algorithms20 Jul

Other references · 11

  1. 2024The Llama 3 Herd of Modelscited by 14,715
  2. 2021Training Verifiers to Solve Math Word Problemscited by 10,109
  3. 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 9,379
  4. 2021Evaluating Large Language Models Trained on Codecited by 8,320
  5. 2021Measuring Mathematical Problem Solving With the MATH Datasetcited by 5,371
  6. 2024Qwen2.5 Technical Reportcited by 4,942
  7. 2018Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actorcited by 4,885
  8. 2015Trust Region Policy Optimizationcited by 2,580
  9. 2024HybridFlow: A Flexible and Efficient RLHF Frameworkcited by 2,057
  10. 2018Exploration by Random Network Distillationcited by 993
  11. 2025One-shot Entropy Minimizationcited by 32

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 12.

12 papers.

  1. 2026Rebellious Student: Reversing Teacher Signals for Reasoning Exploration with Self-Distilled RLVRcited by 15
  2. 2026LaDi-RL: Latent Diffusion Reasoning Prevents Entropy Collapse in Reinforcement Learningcited by 7
  3. 2026From Simulation to Enaction: Post-trained language models recognize and react to their own generationscited by 6
  4. 2026Noisy Data is Destructive to Reinforcement Learning with Verifiable Rewardscited by 5
  5. 2026R2PO: Decoupling Rollout and Inference Policies for LLM Reasoningcited by 3
  6. 2026BandPO: Bridging Trust Regions and Ratio Clipping via Probability-Aware Bounds for LLM Reinforcement Learningcited by 3
  7. 2026Learning at the Right Pace: Adaptive Data Scheduling Improves LLM Reinforcement Learningcited by 2
  8. 2026VAM: Verbalized Action Masking for Controllable Exploration in RL Post-Training -- A Chess Case Studycited by 1
  9. 2026A Unified Framework for Rethinking Policy Divergence Measures in GRPOcited by 0
  10. 2026MAPLE: Modality-Aware Post-training and Learning Ecosystemcited by 0
  11. 2026Don't Let Gains FADE: Breaking Down Policy Gradient Weights in RLcited by 0
  12. 2026GRPODropout: Less is More for Online Reinforcement Learning Rolloutscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026Flexible Entropy Control in RLVR with a Gradient-Preserving Perspective
  2. 2025Revisiting Entropy in Reinforcement Learning for Large Reasoning Models
  3. 2025Exploration vs Exploitation: Rethinking RLVR through Clipping, Entropy, and Spurious Reward
  4. 2025Rethinking Entropy Interventions in RLVR: An Entropy Change Perspective
  5. 2026Understanding and Preventing Entropy Collapse in RLVR with On-Policy Entropy Flow Optimization
  6. 2026Asymmetric Advantage Modulation Calibrates Entropy Dynamics in RLVR
  7. 2025Entropy Ratio Clipping as a Soft Global Constraint for Stable Reinforcement Learning
  8. 2026What are Key Factors for Updates in RL for LLM Reasoning?