mlsanityData to 7 Oct 2026

Understanding and Preventing Entropy Collapse in RLVR with On-Policy Entropy Flow Optimization

arXiv 2605.11491 ↗First posted 12 May 2026cs.LG cs.AIcited by 3

On the map: LLM reasoning → Reinforcement learning for reasoning

Abstract (arXiv)

Reinforcement learning with verifiable rewards (RLVR) has become an effective paradigm for improving the reasoning ability of large language models. However, widely used RLVR algorithms, such as GRPO, often suffer from entropy collapse, leading to premature determinism and unstable optimization. Existing remedies, including entropy regularization and ratio-based clipping heuristics, either control entropy in a coarse-grained manner or rely on approximate on-policy training. In this paper, we revisit entropy collapse from a token-level entropy flow perspective. Our analysis reveals that entropy-decreasing tokens consistently outweigh entropy-increasing ones, resulting in a severely imbalanced entropy flow. This perspective provides a unified explanation of entropy collapse in existing RLVR algorithms and highlights the importance of balancing entropy dynamics. Motivated by this analysis, we propose On-Policy Entropy Flow Optimization (OPEFO), an adaptive entropy flow balancing mechanism that rescales entropy-increasing and entropy-decreasing updates according to their contributions to entropy change, while remaining strict on-policy. Experiments on six mathematical reasoning benchmarks demonstrate that OPEFO improves training stability and final performance. We will release the code and models upon publication.

Key facts. Cited by 3 papers in the map (data through 7 Oct 2026). Placed in Reinforcement learning for reasoning, one of 17 directions in LLM reasoning. Builds on 18 of its 27 references that are in the map. First posted on arXiv 12 May 2026.

Builds on

18 of 27 references.

Closest ancestors · 8

Same direction, newest first

  1. 2025Rethinking Entropy Interventions in RLVR: An Entropy Change Perspective11 Oct
  2. 2025Prosperity before Collapse: How Far Can Off-Policy RL Reach with Stale Data on LLMs?1 Oct
  3. 2025Decomposing the Entropy-Performance Exchange: The Missing Keys to Unlocking Effective Reinforcement Learning4 Aug
  4. 2025Stabilizing Knowledge, Promoting Reasoning: Dual-Token Constraints for RLVR21 Jul
  5. 2025Reasoning with Exploration: An Entropy Perspective17 Jun
  6. 2025The Surprising Effectiveness of Negative Reinforcement in LLM Reasoning2 Jun
  7. 2025On-Policy RL with Optimal Reward Baseline29 May
  8. 2017Proximal Policy Optimization Algorithms20 Jul

Other references · 10

  1. 2021Measuring Mathematical Problem Solving With the MATH Datasetcited by 5,203
  2. 2018Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actorcited by 4,635
  3. 2016Asynchronous Methods for Deep Reinforcement Learningcited by 2,218
  4. 2024HybridFlow: A Flexible and Efficient RLHF Frameworkcited by 1,980
  5. 2017Accurate, Large Minibatch SGD: Training ImageNet in 1 Hourcited by 1,479
  6. 2022Solving Quantitative Reasoning Problems with Language Modelscited by 1,359
  7. 2024OlympiadBench: A Challenging Benchmark for Promoting AGI with Olympiad-Level Bilingual Multimodal Scientific Problemscited by 1,099
  8. 2017A Bayesian Perspective on Generalization and Stochastic Gradient Descentcited by 134
  9. 2025On Entropy Control in LLM-RL Algorithmscited by 39
  10. 2025When Maximum Entropy Misleads Policy Optimizationcited by 7

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2026: 3.

3 papers.

  1. 2026Understanding Evolution Strategies for LLM Reasoning: Broader Reasoning Coverage than GRPOcited by 1
  2. 2026EAPO: Entropy-Driven Adaptive Positive-Negative Sample Weighting for Policy Optimization in Open-Ended QAcited by 0
  3. 2026COEVO: Co-Evolving Context and Parameters for Recursive Self-Improvementcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Revisiting Entropy in Reinforcement Learning for Large Reasoning Models
  2. 2026Flexible Entropy Control in RLVR with a Gradient-Preserving Perspective
  3. 2026Addressing Performance Saturation for LLM RL via Precise Entropy Curve Control
  4. 2025Clip-Low Increases Entropy and Clip-High Decreases Entropy in Reinforcement Learning of Large Language Models
  5. 2026Asymmetric Advantage Modulation Calibrates Entropy Dynamics in RLVR
  6. 2025The Entropy Mechanism of Reinforcement Learning for Reasoning Language Models
  7. 2026PAEC: Position-Aware Entropy Calibration for LLM Reasoning in RLVR
  8. 2026HEALing Entropy Collapse: Enhancing Exploration in Few-Shot RLVR via Hybrid-Domain Entropy Dynamics Alignment