mlsanityData to 14 Oct 2026

EntroPIC: Towards Stable Long-Term Training of LLMs via Entropy Stabilization with Proportional-Integral Control

arXiv 2511.15248 ↗First posted 19 Nov 2025cs.LG cs.AIcited by 13

Not on the map. Why?

Abstract (arXiv)

Long-term training of large language models (LLMs) requires maintaining stable exploration to prevent the model from collapsing into sub-optimal behaviors. Entropy is crucial in this context, as it controls exploration and helps avoid premature convergence to sub-optimal solutions. However, existing reinforcement learning methods struggle to maintain an appropriate level of entropy, as the training process involves a mix of positive and negative samples, each affecting entropy in different ways across steps. To address this, we propose Entropy stabilization via Proportional-Integral Control (EntroPIC), a novel method that adaptively adjusts the influence of positive and negative samples by dynamically tuning their loss coefficients. This approach stabilizes entropy throughout training, ensuring efficient exploration and steady progress. We provide a comprehensive theoretical analysis for both on-policy and off-policy learning settings, demonstrating that EntroPIC is effective at controlling entropy in large-scale LLM training. Experimental results show that our method successfully maintains desired entropy levels, enabling stable and optimal RL training for LLMs.

Key facts. Cited by 13 papers in the map (data through 14 Oct 2026). Builds on 43 of its 53 references that are in the map. First posted on arXiv 19 Nov 2025.

Builds on

43 of 53 references.

References · 43

  1. 2023GPT-4 Technical Reportcited by 18,396
  2. 2022Training language models to follow instructions with human feedbackcited by 15,971
  3. 2017Proximal Policy Optimization Algorithmscited by 15,215
  4. 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 9,379
  5. 2025Qwen3 Technical Reportcited by 8,820
  6. 2023Efficient Memory Management for Large Language Model Serving with PagedAttentioncited by 5,944
  7. 2018Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actorcited by 4,885
  8. 2024GPT-4o System Cardcited by 4,588
  9. 2023Let's Verify Step by Stepcited by 3,789
  10. 2024Gemini 1.5: Unlocking multimodal understanding across millions of tokens of contextcited by 3,224
  11. 2023GPQA: A Graduate-Level Google-Proof Q&A Benchmarkcited by 2,879
  12. 2025DAPO: An Open-Source LLM Reinforcement Learning System at Scalecited by 2,810
31 more
  1. 2016Asynchronous Methods for Deep Reinforcement Learningcited by 2,258
  2. 2024LiveCodeBench: Holistic and Contamination Free Evaluation of Large Language Models for Codecited by 2,004
  3. 2024MMLU-Pro: A More Robust and Challenging Multi-Task Language Understanding Benchmarkcited by 1,848
  4. 2024OlympiadBench: A Challenging Benchmark for Promoting AGI with Olympiad-Level Bilingual Multimodal Scientific Problemscited by 1,136
  5. 2024Tulu 3: Pushing Frontiers in Open Language Model Post-Trainingcited by 1,053
  6. 2018Exploration by Random Network Distillationcited by 993
  7. 2017Reinforcement Learning with Deep Energy-Based Policiescited by 669
  8. 2025The Entropy Mechanism of Reinforcement Learning for Reasoning Language Modelscited by 485
  9. 2025Open-Reasoner-Zero: An Open Source Approach to Scaling Up Reinforcement Learning on the Base Modelcited by 461
  10. 2025MathArena: Evaluating LLMs on Uncontaminated Math Competitionscited by 297
  11. 2023Pretraining Language Models with Human Preferencescited by 247
  12. 2021Maximum Entropy RL (Provably) Solves Some Robust RL Problemscited by 176
  13. 2025AIMO-2 Winning Solution: Building State-of-the-Art Mathematical Reasoning Models with OpenMathReasoning datasetcited by 174
  14. 2025Skywork Open Reasoner 1 Technical Reportcited by 165
  15. 2025The Surprising Effectiveness of Negative Reinforcement in LLM Reasoningcited by 162
  16. 2024Entropy Law: The Story Behind Data Compression and LLM Performancecited by 44
  17. 2025QuestA: Expanding Reasoning Capacity in LLMs via Question Augmentationcited by 44
  18. 2019Maximum Entropy-Regularized Multi-Goal Reinforcement Learningcited by 40
  19. 2024Entropy-Regularized Process Reward Modelcited by 38
  20. 2024Exploration and Anti-Exploration with Distributional Random Network Distillationcited by 33
  21. 2023Flipping Coins to Estimate Pseudocounts for Exploration in Reinforcement Learningcited by 28
  22. 2025CE-GPPO: Coordinating Entropy via Gradient-Preserving Clipping Policy Optimization in Reinforcement Learningcited by 23
  23. 2023Accelerating Reinforcement Learning with Value-Conditional State Entropy Explorationcited by 22
  24. 2019Soft Policy Gradient Method for Maximum Entropy Deep Reinforcement Learningcited by 20
  25. 2025Entropy-guided sequence weighting for efficient exploration in RL-based LLM fine-tuningcited by 13
  26. 2025UloRL:An Ultra-Long Output Reinforcement Learning Approach for Advancing Large Language Models' Reasoning Abilitiescited by 11
  27. 2023The Stable Entropy Hypothesis and Entropy-Aware Decoding: An Analysis and Algorithm for Robust Natural Language Generationcited by 8
  28. 2025A Simple "Motivation" Can Enhance Reinforcement Finetuning of Large Reasoning Modelscited by 7
  29. 2025Supervised Optimism Correction: Be Confident When LLMs Are Surecited by 5
  30. 2025Thinking-Free Policy Initialization Makes Distilled Reasoning Models More Effective and Efficient Reasonerscited by 5
  31. 2024Unused information in token probability distribution of generative LLM: improving LLM reading comprehension through calculation of expected valuescited by 4

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 2; 2026: 11.

13 papers.

Sort:

By area:

  1. 2025AdaptVision: Efficient Vision-Language Models via Adaptive Visual Acquisitioncited by 21
  2. 2025d-TreeRPO: Towards More Reliable Policy Optimization for Diffusion Language Modelscited by 17
  3. 2026On-policy Distillation with Verifiable Rewardcited by 15
  4. 2026STAPO: Stabilizing Reinforcement Learning for LLMs by Silencing Rare Spurious Tokenscited by 9
  5. 2026Composition-RL: Compose Your Verifiable Prompts for Reinforcement Learning of Large Language Modelscited by 6
  6. 2026DemoPSD: Disagreement-Modulated Policy Self-Distillationcited by 6
  7. 2026Intern-S2-Preview: Scientific Agentic Foundation Modelcited by 4
  8. 2026STARE: Surprisal-Guided Token-Level Advantage Reweighting for Policy Entropy Stabilitycited by 3
  9. 2026Group Entropy-Controlled Policy Optimizationcited by 3
  10. 2026Boosting LLM Reasoning via Human-Inspired Reward Shapingcited by 1
  11. 2026Addressing Performance Saturation for LLM RL via Precise Entropy Curve Controlcited by 1
  12. 2026Read the Trace, Steer the Path: Trajectory-Aware Reinforcement Learning for Diffusion Language Modelscited by 0
1 more
  1. 2026Stable-MM-R1: Anchoring Multimodal Reasoning Dynamics via Entropy-Guided Stratificationcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026A Comparative Theoretical Analysis of Entropy Control Methods in Reinforcement Learning
  2. 2026Entropy-Preserving Reinforcement Learning
  3. 2026On the Entropy Dynamics in Reinforcement Fine-Tuning of Large Language Models
  4. 2025SCOPE-RL: Stable and Quantitative Control of Policy Entropy in RL Post-Training
  5. 2026Flexible Entropy Control in RLVR with a Gradient-Preserving Perspective
  6. 2025ESPO: Entropy Importance Sampling Policy Optimization
  7. 2025Revisiting Entropy in Reinforcement Learning for Large Reasoning Models
  8. 2026Entropy-Gated Selective Policy Optimization:Token-Level Gradient Allocation for Hybrid Training of Large Language Models