mlsanityData to 7 Oct 2026

Addressing Performance Saturation for LLM RL via Precise Entropy Curve Control

arXiv 2604.26326 ↗First posted 29 Apr 2026cs.LG cs.CL stat.MLcited by 0

On the map: LLM reasoning → Reinforcement learning for reasoning

Abstract (arXiv)

Reinforcement learning (RL) has enabled complex reasoning abilities in large language models (LLMs). However, most RL algorithms suffer from performance saturation, preventing continued gains as RL training scales. This problem can be characterized by the collapse of entropy, a key diagnostic for exploration in RL. Existing attempts focus on preventing entropy collapse through regularization or clipping. However, their resulting entropy curves often exhibit instability in the long term, which hinders performance gains. In this paper, we introduce Entrocraft, a simple rejection-sampling approach that realizes user-customized entropy schedule by biasing the advantage distributions. Entrocraft requires no objective regularization and is advantage-estimator-agnostic. Theoretically, we relate per-step entropy change to the advantage distribution under minimal assumptions. This explains the behavior of existing RL and entropy-preserving methods. Entrocraft also enables a systematic study of entropy schedules, which reveals that linear annealing, which starts high and decays to a slightly lower target, performs best. Empirically, Entrocraft addresses performance saturation, significantly improving generalization, output diversity, and long-term training. It enables a 4B model to outperform an 8B baseline, sustains improvement for up to 4x longer before plateauing, and raises pass@K by 50% over the baseline.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Placed in Reinforcement learning for reasoning, one of 17 directions in LLM reasoning. Builds on 41 of its 50 references that are in the map. First posted on arXiv 29 Apr 2026.

Builds on

41 of 50 references.

Closest ancestors · 14

Same direction, newest first

  1. 2026Entropy-Preserving Reinforcement Learning12 Mar
  2. 2026Training Reasoning Models on Saturated Problems via Failure-Prefix Conditioning28 Jan
  3. 2025Reasoning with Sampling: Your Base Model is Smarter Than You Think16 Oct
  4. 2025Sample More to Think Less: Group Filtered Policy Optimization for Concise Reasoning13 Aug
  5. 2025The Surprising Effectiveness of Negative Reinforcement in LLM Reasoning2 Jun
  6. 2025The Entropy Mechanism of Reinforcement Learning for Reasoning Language Models28 May
  7. 2025Does Reinforcement Learning Really Incentivize Reasoning Capacity in LLMs Beyond the Base Model?18 Apr
  8. 2025A Minimalist Approach to LLM Reasoning: from Rejection Sampling to Reinforce15 Apr
  9. 2025Echo Chamber: RL Post-training Amplifies Behaviors Learned in Pretraining10 Apr
  10. 2025Right Question is Already Half the Answer: Fully Unsupervised LLM Reasoning Incentivization8 Apr
  11. 2025Understanding R1-Zero-Like Training: A Critical Perspective26 Mar
  12. 2025DAPO: An Open-Source LLM Reinforcement Learning System at Scale18 Mar
2 more
  1. 2023RAFT: Reward rAnked FineTuning for Generative Foundation Model Alignment13 Apr
  2. 2017Proximal Policy Optimization Algorithms20 Jul

Other references · 27

  1. 2022Training language models to follow instructions with human feedbackcited by 15,712
  2. 2024The Llama 3 Herd of Modelscited by 14,192
  3. 2021Training Verifiers to Solve Math Word Problemscited by 9,743
  4. 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 8,851
  5. 2025Qwen3 Technical Reportcited by 8,195
  6. 2021Evaluating Large Language Models Trained on Codecited by 8,053
  7. 2023Efficient Memory Management for Large Language Model Serving with PagedAttentioncited by 5,767
  8. 2021Measuring Mathematical Problem Solving With the MATH Datasetcited by 5,203
  9. 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedbackcited by 3,634
  10. 2016Asynchronous Methods for Deep Reinforcement Learningcited by 2,218
  11. 2024HybridFlow: A Flexible and Efficient RLHF Frameworkcited by 1,980
  12. 2024LiveCodeBench: Holistic and Contamination Free Evaluation of Large Language Models for Codecited by 1,906
15 more
  1. 2025Group Sequence Policy Optimizationcited by 788
  2. 2017Reinforcement Learning with Deep Energy-Based Policiescited by 659
  3. 2023PyTorch FSDP: Experiences on Scaling Fully Sharded Data Parallelcited by 641
  4. 2024Learning Dynamics of LLM Finetuningcited by 93
  5. 2025Challenging the Boundaries of Reasoning: An Olympiad-Level Math Benchmark for Large Language Modelscited by 65
  6. 2025Horizon Reduction Makes RL Scalablecited by 55
  7. 2025On Entropy Control in LLM-RL Algorithmscited by 39
  8. 2025Stabilizing Reinforcement Learning with LLMs: Formulation and Practicescited by 38
  9. 2024Does RLHF Scale? Exploring the Impacts From Data, Model, and Methodcited by 22
  10. 2025Benchmarking Music Generation Models and Metrics via Human Preference Studiescited by 15
  11. 2025EPO: Entropy-regularized Policy Optimization for LLM Agents Reinforcement Learningcited by 15
  12. 2025EntroPIC: Towards Stable Long-Term Training of LLMs via Entropy Stabilization with Proportional-Integral Controlcited by 13
  13. 2026On the Entropy Dynamics in Reinforcement Fine-Tuning of Large Language Modelscited by 8
  14. 2024It Takes Two: On the Seamlessness between Reward and Policy Model in RLHFcited by 5
  15. 2026Preventing Learning Stagnation in PPO by Scaling to 1 Million Parallel Environmentscited by 3

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2025Revisiting Entropy in Reinforcement Learning for Large Reasoning Models
  2. 2025Rethinking Entropy Interventions in RLVR: An Entropy Change Perspective
  3. 2026Flexible Entropy Control in RLVR with a Gradient-Preserving Perspective
  4. 2026A Comparative Theoretical Analysis of Entropy Control Methods in Reinforcement Learning
  5. 2025Revisiting Entropy Regularization: Adaptive Coefficient Unlocks Its Potential for LLM Reinforcement Learning
  6. 2026Understanding and Preventing Entropy Collapse in RLVR with On-Policy Entropy Flow Optimization
  7. 2026Asymmetric Advantage Modulation Calibrates Entropy Dynamics in RLVR
  8. 2025Efficient Reinforcement Learning with Semantic and Token Entropy for LLM Reasoning