# Entropy-Adaptive Fine-Tuning: Resolving Confident Conflicts to Mitigate Forgetting

arXiv: https://arxiv.org/abs/2601.02151 · HTML: https://mlsanity.com/paper/2601.02151/

## Key facts

- Cited by 16 papers in the map (data through 14 Oct 2026).
- Builds on 24 of its 36 references that are in the map.
- First posted on arXiv 5 Jan 2026.

## Builds on (most cited first, 24 of 24)

- [Training language models to follow instructions with human feedback](https://mlsanity.com/paper/2203.02155/) (2022)
- [Proximal Policy Optimization Algorithms](https://mlsanity.com/paper/1707.06347/) (2017)
- [Measuring Massive Multitask Language Understanding](https://mlsanity.com/paper/2009.03300/) (2020)
- [Overcoming catastrophic forgetting in neural networks](https://mlsanity.com/paper/1612.00796/) (2016)
- [The Curious Case of Neural Text Degeneration](https://mlsanity.com/paper/1904.09751/) (2019)
- [Learning without Forgetting](https://mlsanity.com/paper/1606.09282/) (2016)
- [Gradient Episodic Memory for Continual Learning](https://mlsanity.com/paper/1706.08840/) (2017)
- [LlamaFactory: Unified Efficient Fine-Tuning of 100+ Language Models](https://mlsanity.com/paper/2403.13372/) (2024)
- [What Disease does this Patient Have? A Large-scale Open Domain Question Answering Dataset from Medical Exams](https://mlsanity.com/paper/2009.13081/) (2020)
- [PubMedQA: A Dataset for Biomedical Research Question Answering](https://mlsanity.com/paper/1909.06146/) (2019)
- [Hierarchical Neural Story Generation](https://mlsanity.com/paper/1805.04833/) (2018)
- [ChatGLM: A Family of Large Language Models from GLM-130B to GLM-4 All Tools](https://mlsanity.com/paper/2406.12793/) (2024)
- [SFT Memorizes, RL Generalizes: A Comparative Study of Foundation Model Post-training](https://mlsanity.com/paper/2501.17161/) (2025)
- [An Empirical Study of Catastrophic Forgetting in Large Language Models During Continual Fine-tuning](https://mlsanity.com/paper/2308.08747/) (2023)
- [SimpleRL-Zoo: Investigating and Taming Zero Reinforcement Learning for Open Base Models in the Wild](https://mlsanity.com/paper/2503.18892/) (2025)
- [CLUE: A Chinese Language Understanding Evaluation Benchmark](https://mlsanity.com/paper/2004.05986/) (2020)
- [HuatuoGPT-o1, Towards Medical Complex Reasoning with LLMs](https://mlsanity.com/paper/2412.18925/) (2024)
- [Continual Learning of Large Language Models: A Comprehensive Survey](https://mlsanity.com/paper/2404.16789/) (2024)
- [Lingshu: A Generalist Foundation Model for Unified Multimodal Medical Understanding and Reasoning](https://mlsanity.com/paper/2506.07044/) (2025)
- [On the Generalization of SFT: A Reinforcement Learning Perspective with Reward Rectification](https://mlsanity.com/paper/2508.05629/) (2025)
- [Nemotron-CrossThink: Scaling Self-Learning beyond Math Reasoning](https://mlsanity.com/paper/2504.13941/) (2025)
- [Vanishing Gradients in Reinforcement Finetuning of Language Models](https://mlsanity.com/paper/2310.20703/) (2023)
- [Implicit Reward as the Bridge: A Unified View of SFT and DPO Connections](https://mlsanity.com/paper/2507.00018/) (2025)
- [Upweighting Easy Samples in Fine-Tuning Mitigates Forgetting](https://mlsanity.com/paper/2502.02797/) (2025)

## Built on by (most cited first, 16 of 16)

- [ProFit: Leveraging High-Value Signals in SFT via Probability-Guided Token Selection](https://mlsanity.com/paper/2601.09195/) (2026)
- [Good SFT Optimizes for SFT, Better SFT Prepares for Reinforcement Learning](https://mlsanity.com/paper/2602.01058/) (2026)
- [Towards On-Policy SFT: Distribution Discriminant Theory and its Applications in LLM Training](https://mlsanity.com/paper/2602.12222/) (2026)
- [Probability-Entropy Calibration: An Elastic Indicator for Adaptive Fine-tuning](https://mlsanity.com/paper/2602.01745/) (2026)
- [Gradients Must Earn Their Influence: Unifying SFT with Generalized Entropic Objectives](https://mlsanity.com/paper/2602.11424/) (2026)
- [PEFT-Arena: Understanding Parameter-Efficient Finetuning from a Stability-Plasticity Perspective](https://mlsanity.com/paper/2605.28819/) (2026)
- [PriFT: Prior-Support Guided Supervised Fine-Tuning](https://mlsanity.com/paper/2606.09396/) (2026)
- [LP-SFT: Keeping Plausible Alternatives Alive in Supervised Fine-Tuning](https://mlsanity.com/paper/2607.04733/) (2026)
- [CPR for LLMs: Critical-Point Routing against Catastrophic Forgetting in Domain Adaptation](https://mlsanity.com/paper/2608.30158/) (2026)
- [Supervised Fine-Tuning Needs to Unlock the Potential of Token Priority](https://mlsanity.com/paper/2602.01227/) (2026)
- [A Unifying Lens on Supervised Fine-Tuning Through Target Distribution Design](https://mlsanity.com/paper/2606.11189/) (2026)
- [Fine-Tuning LLMs for Translation: General Forgetting Mitigation Does Not Preserve MT-Specific Instruction Following](https://mlsanity.com/paper/2609.28395/) (2026)
- [Rethinking Token Reweighting for SFT: Suppress, Reverse, and Extrapolate Learned Features](https://mlsanity.com/paper/2609.33463/) (2026)
- [Learning Dynamics of Continual Learning: A Unified View of Data Attribution, Forgetting, and Plasticity Loss](https://mlsanity.com/paper/2609.33620/) (2026)
- [Learning What to Imitate: Entropy-Aware Distribution Mixing](https://mlsanity.com/paper/2610.06671/) (2026)
- [Beyond Distribution Matching: Selective Margin Preservation for Supervised Fine-Tuning](https://mlsanity.com/paper/2610.19685/) (2026)

## Abstract (arXiv)

Supervised Fine-Tuning (SFT) is the standard paradigm for domain adaptation, yet it frequently incurs the cost of catastrophic forgetting. In sharp contrast, on-policy Reinforcement Learning (RL) effectively preserves general capabilities. We investigate this discrepancy and identify a fundamental distributional gap: while RL aligns with the model's internal belief, SFT forces the model to fit external supervision. This mismatch often manifests as "Confident Conflicts" tokens characterized by low probability but low entropy. In these instances, the model is highly confident in its own prediction but is forced to learn a divergent ground truth, triggering destructive gradient updates. To address this, we propose Entropy-Adaptive Fine-Tuning (EAFT). Unlike methods relying solely on prediction probability, EAFT utilizes token-level entropy as a gating mechanism to distinguish between epistemic uncertainty and knowledge conflict. This allows the model to learn from uncertain samples while suppressing gradients on conflicting data. Extensive experiments on Qwen and GLM series (ranging from 4B to 32B parameters) across mathematical, medical, and agentic domains confirm our hypothesis. EAFT consistently matches the downstream performance of standard SFT while significantly mitigating the degradation of general capabilities.