mlsanityData to 14 Oct 2026

Mitigating Forgetting in LLM Fine-Tuning via Low-Perplexity Token Learning

arXiv 2501.14315 ↗First posted 24 Jan 2025cs.CLcited by 14

On the map: Synthetic data and self-improvement for LLMs → Self-rewarding alignment and self-training

Abstract (arXiv)

Maintaining consistent model performance across domains is a fundamental challenge in machine learning. While recent work has explored using LLM-generated data for fine-tuning, its impact on cross-domain generalization remains poorly understood. This paper presents a systematic analysis revealing that fine-tuning with LLM-generated data not only improves target task performance but also reduces non-target task degradation compared to fine-tuning with ground truth data. Through analyzing the data sequence in tasks of various domains, we demonstrate that this enhancement of non-target task robustness stems from the reduction of high perplexity tokens found in LLM-generated sequences. Following our findings, we showed that masking high perplexity tokens in ground truth training data achieves similar non-target task performance preservation, comparable to using LLM-generated data. Extensive experiments across different model families and scales, including Gemma 2 IT 2B, Llama 3 8B Instruct, and three additional models, agree with our findings. To the best of our knowledge, this is the first work to provide an empirical explanation based on token perplexity reduction to mitigate catastrophic forgetting in LLMs after fine-tuning, offering valuable insights for developing more robust fine-tuning strategies.

Key facts. Cited by 14 papers in the map (data through 14 Oct 2026). Placed in Self-rewarding alignment and self-training, one of 13 directions in Synthetic data and self-improvement for LLMs. Builds on 45 of its 57 references that are in the map. First posted on arXiv 24 Jan 2025.

Builds on

45 of 57 references.

Closest ancestors · 2

Same direction, newest first

  1. 2024Magpie: Alignment Data Synthesis from Scratch by Prompting Aligned LLMs with Nothing12 Jun
  2. 2023Reinforced Self-Training (ReST) for Language Modeling17 Aug

Other references · 43

  1. 2023GPT-4 Technical Reportcited by 18,396
  2. 2022Training language models to follow instructions with human feedbackcited by 15,971
  3. 2024The Llama 3 Herd of Modelscited by 14,715
  4. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,380
  5. 2021Training Verifiers to Solve Math Word Problemscited by 10,109
  6. 2021Measuring Mathematical Problem Solving With the MATH Datasetcited by 5,371
  7. 2018Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challengecited by 4,886
  8. 2021Finetuned Language Models Are Zero-Shot Learnerscited by 3,780
  9. 2021Program Synthesis with Large Language Modelscited by 3,637
  10. 2023Mistral 7Bcited by 3,164
  11. 2023Universal and Transferable Adversarial Attacks on Aligned Language Modelscited by 3,104
  12. 2022Scaling Instruction-Finetuned Language Modelscited by 3,024
31 more
  1. 2022Self-Instruct: Aligning Language Models with Self-Generated Instructionscited by 2,724
  2. 2020Learning to summarize from human feedbackcited by 2,588
  3. 2024Gemma 2: Improving Open Language Models at a Practical Sizecited by 1,998
  4. 2021WebGPT: Browser-assisted question-answering with human feedbackcited by 1,745
  5. 2023Instruction-Following Evaluation for Large Language Modelscited by 1,228
  6. 2023Fine-tuning Aligned Language Models Compromises Safety, Even When Users Do Not Intend To!cited by 1,154
  7. 2023WizardLM: Empowering large pre-trained language models to follow complex instructionscited by 1,125
  8. 2023LIMA: Less Is More for Alignmentcited by 1,100
  9. 2022STaR: Bootstrapping Reasoning With Reasoningcited by 940
  10. 2024DoRA: Weight-Decomposed Low-Rank Adaptationcited by 758
  11. 2023OpenAssistant Conversations -- Democratizing Large Language Model Alignmentcited by 721
  12. 2023Can LLM Already Serve as A Database Interface? A BIg Bench for Large-Scale Database Grounded Text-to-SQLscited by 688
  13. 2023How Far Can Camels Go? Exploring the State of Instruction Tuning on Open Resourcescited by 408
  14. 2023Beyond Human Data: Scaling Self-Training for Problem-Solving with Language Modelscited by 349
  15. 2024LoRA Learns Less and Forgets Lesscited by 328
  16. 2022Prioritized Training on Points that are Learnable, Worth Learning, and Not Yet Learntcited by 236
  17. 2024Mitigating Catastrophic Forgetting in Large Language Models with Self-Synthesized Rehearsalcited by 118
  18. 2024Self-Distillation Bridges Distribution Gap in Language Model Fine-Tuningcited by 96
  19. 2023Bootstrap Your Own Skills: Learning to Solve New Tasks with Large Language Model Guidancecited by 86
  20. 2024Perplexed by Perplexity: Perplexity-Based Data Pruning With Small Reference Modelscited by 84
  21. 2024A Closer Look at the Limitations of Instruction Tuningcited by 77
  22. 2024LLM Discussion: Enhancing the Creativity of Large Language Models via Discussion Framework and Role-Playcited by 66
  23. 2024StreamBench: Towards Benchmarking Continuous Improvement of Language Agentscited by 47
  24. 2020Leverage the Average: an Analysis of KL Regularization in RLcited by 33
  25. 2024Fine-Tuning is Fine, if Calibratedcited by 24
  26. 2024I Learn Better If You Speak My Language: Understanding the Superior Performance of Fine-Tuning Large Language Models with LLM-Generated Responsescited by 15
  27. 2024Synthesizing Programmatic Reinforcement Learning Policies with Large Language Model Guided Searchcited by 13
  28. 2022CodexDB: Generating Code for Processing SQL Queries using GPT-3 Codexcited by 5
  29. 2024SimTube: Generating Simulated Video Comments through Multimodal AI and User Personascited by 5
  30. 2023Location-Aware Visual Question Generation with Lightweight Modelscited by 4
  31. 2024Selective Self-Rehearsal: A Fine-Tuning Approach to Improve Generalization in Large Language Modelscited by 1

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 1; 2026: 13.

14 papers.

Sort:

By area:

  1. 2026Cornerstones or Stumbling Blocks? Deciphering the Rock Tokens in On-Policy Distillationcited by 11
  2. 2025Think Consistently, Reason Efficiently: Energy-Based Calibration for Implicit Chain-of-Thoughtcited by 4
  3. 2026Gradients Must Earn Their Influence: Unifying SFT with Generalized Entropic Objectivescited by 3
  4. 2026OptProver: Bridging Olympiad and Optimization through Continual Training in Formal Theorem Provingcited by 1
  5. 2026Stabilizing LLM Supervised Fine-Tuning via Explicit Distributional Controlcited by 1
  6. 2026Fine-Tuning Without Forgetting via Loss-Adaptive Learning Ratescited by 1
  7. 2026AlphaToken: Decoupling Adaptation and Stability for Path-Aware Response Token Valuation in LLM Post-Trainingcited by 1
  8. 2026One Algorithm, Two Goals: Dual Scoring for Parameter and Data Selection in LLM Fine-Tuningcited by 0
  9. 2026SCOReD: Student-Aware CoT Optimization for Recommendation Distillationcited by 0
  10. 2026Knowing but Not Saying: Preventing Factual Access Failures in LLM SFT via Recall-Anchored Distillationcited by 0
  11. 2026The Safeguard Worked. Is the LLM System Safer?cited by 0
  12. 2026Fine-Tuning LLMs for Translation: General Forgetting Mitigation Does Not Preserve MT-Specific Instruction Followingcited by 0
2 more
  1. 2026Learn Here, Move Less Elsewhere: Input-Conditioned Plasticity from Retained-Domain Activation Atlasescited by 0
  2. 2026Geometric Calibration of Supervised Fine-Tuningcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026CapTrack: Multifaceted Evaluation of Forgetting in LLM Post-Training
  2. 2024Scaling Laws for Forgetting When Fine-Tuning Large Language Models
  3. 2025Leaner Training, Lower Leakage: Revisiting Memorization in LLM Fine-Tuning with LoRA
  4. 2026Quantization-Robust Unlearning through the Lens of Retain-Forget Loss Landscapes Interaction
  5. 2024Understanding Forgetting in LLM Supervised Fine-Tuning and Preference Learning -- A Convex Optimization Perspective
  6. 2025Catastrophic Forgetting in LLMs: A Comparative Analysis Across Language Tasks
  7. 2025Unveiling Over-Memorization in Finetuning LLMs for Reasoning Tasks
  8. 2025Unlearning That Lasts: Utility-Preserving, Robust, and Almost Irreversible Forgetting in LLMs