# On the Entropy Dynamics in Reinforcement Fine-Tuning of Large Language Models

arXiv: https://arxiv.org/abs/2602.03392 · HTML: https://mlsanity.com/paper/2602.03392/

## Key facts

- Cited by 8 papers in the map (data through 14 Oct 2026).
- Builds on 23 of its 32 references that are in the map.
- First posted on arXiv 3 Feb 2026.

## Builds on (most cited first, 23 of 23)

- [Proximal Policy Optimization Algorithms](https://mlsanity.com/paper/1707.06347/) (2017)
- [DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models](https://mlsanity.com/paper/2402.03300/) (2024)
- [Qwen3 Technical Report](https://mlsanity.com/paper/2505.09388/) (2025)
- [Qwen2.5 Technical Report](https://mlsanity.com/paper/2412.15115/) (2024)
- [Let's Verify Step by Step](https://mlsanity.com/paper/2305.20050/) (2023)
- [DAPO: An Open-Source LLM Reinforcement Learning System at Scale](https://mlsanity.com/paper/2503.14476/) (2025)
- [gpt-oss-120b & gpt-oss-20b Model Card](https://mlsanity.com/paper/2508.10925/) (2025)
- [Group Sequence Policy Optimization](https://mlsanity.com/paper/2507.18071/) (2025)
- [Beyond the 80/20 Rule: High-Entropy Minority Tokens Drive Effective Reinforcement Learning for LLM Reasoning](https://mlsanity.com/paper/2506.01939/) (2025)
- [The Entropy Mechanism of Reinforcement Learning for Reasoning Language Models](https://mlsanity.com/paper/2505.22617/) (2025)
- [A Brief Survey of Deep Reinforcement Learning](https://mlsanity.com/paper/1708.05866/) (2017)
- [GLM-4.5: Agentic, Reasoning, and Coding (ARC) Foundation Models](https://mlsanity.com/paper/2508.06471/) (2025)
- [Open-Reasoner-Zero: An Open Source Approach to Scaling Up Reinforcement Learning on the Base Model](https://mlsanity.com/paper/2503.24290/) (2025)
- [ReTool: Reinforcement Learning for Strategic Tool Use in LLMs](https://mlsanity.com/paper/2504.11536/) (2025)
- [SWE-RL: Advancing LLM Reasoning via Reinforcement Learning on Open Software Evolution](https://mlsanity.com/paper/2502.18449/) (2025)
- [Understanding the impact of entropy on policy optimization](https://mlsanity.com/paper/1811.11214/) (2018)
- [Learning Dynamics of LLM Finetuning](https://mlsanity.com/paper/2407.10490/) (2024)
- [Rewarding the Unlikely: Lifting GRPO Beyond Distribution Sharpening](https://mlsanity.com/paper/2506.02355/) (2025)
- [Nemotron-Research-Tool-N1: Exploring Tool-Using Language Models with Reinforced Reasoning](https://mlsanity.com/paper/2505.00024/) (2025)
- [Part I: Tricks or Traps? A Deep Dive into RL for LLM Reasoning](https://mlsanity.com/paper/2508.08221/) (2025)
- [Do Not Let Low-Probability Tokens Over-Dominate in RL for LLMs](https://mlsanity.com/paper/2505.12929/) (2025)
- [Enhancing Efficiency and Exploration in Reinforcement Learning for LLMs](https://mlsanity.com/paper/2505.18573/) (2025)
- [Trinity-RFT: A General-Purpose and Unified Framework for Reinforcement Fine-Tuning of Large Language Models](https://mlsanity.com/paper/2505.17826/) (2025)

## Built on by (most cited first, 8 of 8)

- [Entropy-Preserving Reinforcement Learning](https://mlsanity.com/paper/2603.11682/) (2026)
- [STAPO: Stabilizing Reinforcement Learning for LLMs by Silencing Rare Spurious Tokens](https://mlsanity.com/paper/2602.15620/) (2026)
- [STARE: Surprisal-Guided Token-Level Advantage Reweighting for Policy Entropy Stability](https://mlsanity.com/paper/2606.19236/) (2026)
- [Which Tokens Matter? Adaptive Token Selection for RLVR with the Relative Surprisal Index](https://mlsanity.com/paper/2606.31575/) (2026)
- [Group Entropy-Controlled Policy Optimization](https://mlsanity.com/paper/2607.16850/) (2026)
- [Addressing Performance Saturation for LLM RL via Precise Entropy Curve Control](https://mlsanity.com/paper/2604.26326/) (2026)
- [Headroom-Drift Replay: A Primitive for Principled Replay Control in GRPO](https://mlsanity.com/paper/2609.03941/) (2026)
- [Fine-Tuning Improves Information Conveyance in Language Models](https://mlsanity.com/paper/2605.30844/) (2026)

## Abstract (arXiv)

Entropy serves as a critical metric for measuring the diversity of outputs generated by large language models (LLMs), providing valuable insights into their exploration capabilities. While recent studies increasingly focus on monitoring and adjusting entropy to better balance exploration and exploitation in reinforcement fine-tuning (RFT), a principled understanding of entropy dynamics during this process is yet to be thoroughly investigated. In this paper, we establish a theoretical framework for analyzing the entropy dynamics during the RFT process, which begins with a discriminant expression that quantifies entropy change under a single logit update. This foundation enables the derivation of a first-order expression for entropy change, which can be further extended to the update formula of Group Relative Policy Optimization (GRPO). The corollaries and insights drawn from the theoretical analysis inspire the design of entropy control methods, and also offer a unified lens for interpreting various entropy-based methods in existing studies. We provide empirical evidence to support the main conclusions of our analysis and demonstrate the effectiveness of the derived entropy-discriminator clipping methods. This study yields novel insights into RFT training dynamics, providing theoretical support and practical strategies for optimizing the exploration-exploitation balance during LLM fine-tuning.