mlsanityData to 7 Oct 2026

Entropy-Preserving Reinforcement Learning

arXiv 2603.11682 ↗First posted 12 Mar 2026cs.LG cs.AIcited by 10

On the map: LLM reasoning → Reinforcement learning for reasoning

Abstract (arXiv)

Policy gradient algorithms have driven many recent advancements in language model reasoning. An appealing property is their ability to learn from exploration on their own trajectories, a process crucial for fostering diverse and creative solutions. As we show in this paper, many policy gradient algorithms naturally reduce the entropy -- and thus the diversity of explored trajectories -- as part of training, yielding a policy increasingly limited in its ability to explore. In this paper, we argue that entropy should be actively monitored and controlled throughout training. We formally analyze the contributions of leading policy gradient objectives on entropy dynamics, identify empirical factors (such as numerical precision) that significantly impact entropy behavior, and propose explicit mechanisms for entropy control. These include REPO, a family of algorithms that modify the advantage function to regulate entropy, and ADAPO, an adaptive asymmetric clipping approach. Models trained with our entropy-preserving methods maintain diversity throughout training, yielding final policies that are more performant and retain their trainability for sequential learning in new environments.

Key facts. Cited by 10 papers in the map (data through 7 Oct 2026). Placed in Reinforcement learning for reasoning, one of 17 directions in LLM reasoning. Builds on 43 of its 56 references that are in the map. First posted on arXiv 12 Mar 2026.

Builds on

43 of 56 references.

Closest ancestors · 17

Same direction, newest first

  1. 2025Pass@k Training for Adaptively Balancing Exploration and Exploitation of Large Reasoning Models14 Aug
  2. 2025Spurious Rewards: Rethinking Training Signals in RLVR12 Jun
  3. 2025Rewarding the Unlikely: Lifting GRPO Beyond Distribution Sharpening3 Jun
  4. 2025ProRL: Prolonged Reinforcement Learning Expands Reasoning Boundaries in Large Language Models30 May
  5. 2025The Entropy Mechanism of Reinforcement Learning for Reasoning Language Models28 May
  6. 2025Reinforcement Learning for Reasoning in Large Language Models with One Training Example29 Apr
  7. 2025TTRL: Test-Time Reinforcement Learning22 Apr
  8. 2025Does Reinforcement Learning Really Incentivize Reasoning Capacity in LLMs Beyond the Base Model?18 Apr
  9. 2025Echo Chamber: RL Post-training Amplifies Behaviors Learned in Pretraining10 Apr
  10. 2025Right Question is Already Half the Answer: Fully Unsupervised LLM Reasoning Incentivization8 Apr
  11. 2025Understanding R1-Zero-Like Training: A Critical Perspective26 Mar
  12. 2025DAPO: An Open-Source LLM Reinforcement Learning System at Scale18 Mar
5 more
  1. 2025Cognitive Behaviors that Enable Self-Improving Reasoners, or, Four Habits of Highly Effective STaRs3 Mar
  2. 2025Ignore the KL Penalty! Boosting Exploration on Critical Tokens to Enhance RL Fine-Tuning10 Feb
  3. 2025Kimi k1.5: Scaling Reinforcement Learning with LLMs22 Jan
  4. 2024Tulu 3: Pushing Frontiers in Open Language Model Post-Training22 Nov
  5. 2017Proximal Policy Optimization Algorithms20 Jul

Other references · 26

  1. 2022Training language models to follow instructions with human feedbackcited by 15,712
  2. 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 8,851
  3. 2025Qwen3 Technical Reportcited by 8,195
  4. 2023Efficient Memory Management for Large Language Model Serving with PagedAttentioncited by 5,767
  5. 2018Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actorcited by 4,635
  6. 2025Gemini 2.5: Pushing the Frontier with Advanced Reasoning, Multimodality, Long Context, and Next Generation Agentic Capabilitiescited by 3,655
  7. 2020Learning to summarize from human feedbackcited by 2,562
  8. 2015Trust Region Policy Optimizationcited by 2,503
  9. 2019Fine-Tuning Language Models from Human Preferencescited by 2,124
  10. 2022Language Models (Mostly) Know What They Knowcited by 1,783
  11. 2023On-Policy Distillation of Language Models: Learning from Self-Generated Mistakescited by 840
  12. 2025Group Sequence Policy Optimizationcited by 788
14 more
  1. 2017Reinforcement Learning with Deep Energy-Based Policiescited by 659
  2. 2024AppWorld: A Controllable World of Apps and People for Benchmarking Interactive Coding Agentscited by 359
  3. 2019DD-PPO: Learning Near-Perfect PointGoal Navigators from 2.5 Billion Framescited by 352
  4. 2021Maximum Entropy RL (Provably) Solves Some Robust RL Problemscited by 169
  5. 2022RL with KL penalties is better viewed as Bayesian inferencecited by 130
  6. 2025Reinforcement Learning for Long-Horizon Interactive LLM Agentscited by 103
  7. 2025Defeating the Training-Inference Mismatch via FP16cited by 62
  8. 2024Self-Consistency Preference Optimizationcited by 50
  9. 2025BAPO: Stabilizing Off-Policy Reinforcement Learning for LLMs via Balanced Policy Optimization with Adaptive Clippingcited by 47
  10. 2024Asymptotics of Language Model Alignmentcited by 46
  11. 2024Cut Your Losses in Large-Vocabulary Language Modelscited by 35
  12. 2025Towards Enterprise-Ready Computer Using Generalist Agentcited by 31
  13. 2026On the Entropy Dynamics in Reinforcement Fine-Tuning of Large Language Modelscited by 8
  14. 2024SimpleFSDP: Simpler Fully Sharded Data Parallel with torch.compilecited by 7

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2026: 10.

10 papers.

  1. 2026On-policy Distillation with Verifiable Rewardcited by 10
  2. 2026A Tale of Two Temperatures: Simple, Efficient, and Diverse Sampling from Diffusion Language Modelscited by 5
  3. 2026AEM: Adaptive Entropy Modulation for Multi-Turn Agentic Reinforcement Learningcited by 5
  4. 2026Entropy Polarity in Reinforcement Fine-Tuning: Direction, Asymmetry, and Controlcited by 2
  5. 2026Reinforcement Learning over Predictive Distributions for LLM Regressioncited by 1
  6. 2026Multi-Step Likelihood-Ratio Correction for Reinforcement Learning with Verifiable Rewardscited by 1
  7. 2026Understanding Evolution Strategies for LLM Reasoning: Broader Reasoning Coverage than GRPOcited by 1
  8. 2026Headroom-Drift Replay: A Primitive for Principled Replay Control in GRPOcited by 1
  9. 2026Addressing Performance Saturation for LLM RL via Precise Entropy Curve Controlcited by 0
  10. 2026Dependency-Aware Reward Shaping for Agentic Reinforcement Learningcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025CE-GPPO: Coordinating Entropy via Gradient-Preserving Clipping Policy Optimization in Reinforcement Learning
  2. 2026A Comparative Theoretical Analysis of Entropy Control Methods in Reinforcement Learning
  3. 2026Flexible Entropy Control in RLVR with a Gradient-Preserving Perspective
  4. 2026Targeted Exploration via Unified Entropy Control for Reinforcement Learning
  5. 2025Rethinking Entropy Interventions in RLVR: An Entropy Change Perspective
  6. 2025Harnessing Uncertainty: Entropy-Modulated Policy Gradients for Long-Horizon LLM Agents
  7. 2026Entropy-Gated Selective Policy Optimization:Token-Level Gradient Allocation for Hybrid Training of Large Language Models
  8. 2025ESPO: Entropy Importance Sampling Policy Optimization