mlsanityData to 7 Oct 2026

Self-Distillation Bridges Distribution Gap in Language Model Fine-Tuning

arXiv 2402.13669 ↗First posted 21 Feb 2024cs.CLcited by 90

Not on the map. Why?

Abstract (arXiv)

The surge in Large Language Models (LLMs) has revolutionized natural language processing, but fine-tuning them for specific tasks often encounters challenges in balancing performance and preserving general instruction-following abilities. In this paper, we posit that the distribution gap between task datasets and the LLMs serves as the primary underlying cause. To address the problem, we introduce Self-Distillation Fine-Tuning (SDFT), a novel approach that bridges the distribution gap by guiding fine-tuning with a distilled dataset generated by the model itself to match its original distribution. Experimental results on the Llama-2-chat model across various benchmarks demonstrate that SDFT effectively mitigates catastrophic forgetting while achieving comparable or superior performance on downstream tasks compared to the vanilla fine-tuning. Moreover, SDFT demonstrates the potential to maintain the helpfulness and safety alignment of LLMs. Our code is available at https://github.com/sail-sg/sdft.

Key facts. Cited by 90 papers in the map (data through 7 Oct 2026). Builds on 36 of its 50 references that are in the map. First posted on arXiv 21 Feb 2024.

Builds on

36 of 50 references.

References · 36

  1. 2020Language Models are Few-Shot Learnerscited by 32,199
  2. 2022Training language models to follow instructions with human feedbackcited by 15,712
  3. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  4. 2021Training Verifiers to Solve Math Word Problemscited by 9,743
  5. 2019Sentence-BERT: Sentence Embeddings using Siamese BERT-Networkscited by 9,049
  6. 2021Evaluating Large Language Models Trained on Codecited by 8,053
  7. 2020Measuring Massive Multitask Language Understandingcited by 7,768
  8. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,304
  9. 2016Overcoming catastrophic forgetting in neural networkscited by 4,718
  10. 2018Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challengecited by 4,710
  11. 2019HellaSwag: Can a Machine Really Finish Your Sentence?cited by 4,018
  12. 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedbackcited by 3,634
24 more
  1. 2023Self-Refine: Iterative Refinement with Self-Feedbackcited by 3,317
  2. 2022Scaling Instruction-Finetuned Language Modelscited by 2,997
  3. 2021TruthfulQA: Measuring How Models Mimic Human Falsehoodscited by 2,919
  4. 2023Universal and Transferable Adversarial Attacks on Aligned Language Modelscited by 2,914
  5. 2022Self-Instruct: Aligning Language Models with Self-Generated Instructionscited by 2,674
  6. 2023Code Llama: Open Foundation Models for Codecited by 2,105
  7. 2017Gradient Episodic Memory for Continual Learningcited by 1,871
  8. 2023Gorilla: Large Language Model Connected with Massive APIscited by 1,302
  9. 2023WizardLM: Empowering large pre-trained language models to follow complex instructionscited by 1,116
  10. 2023Fine-tuning Aligned Language Models Compromises Safety, Even When Users Do Not Intend To!cited by 1,106
  11. 2017Memory Aware Synapses: Learning what (not) to forgetcited by 1,094
  12. 2023LIMA: Less Is More for Alignmentcited by 1,080
  13. 2023BloombergGPT: A Large Language Model for Financecited by 923
  14. 2017PackNet: Adding Multiple Tasks to a Single Network by Iterative Pruningcited by 882
  15. 2023WizardCoder: Empowering Code Large Language Models with Evol-Instructcited by 728
  16. 2023An Empirical Study of Catastrophic Forgetting in Large Language Models During Continual Fine-tuningcited by 622
  17. 2023WizardMath: Empowering Mathematical Reasoning for Large Language Models via Reinforced Evol-Instructcited by 618
  18. 2024Self-Play Fine-Tuning Converts Weak Language Models to Strong Language Modelscited by 552
  19. 2023How Far Can Camels Go? Exploring the State of Instruction Tuning on Open Resourcescited by 405
  20. 2023Safety-Tuned LLaMAs: Lessons From Improving the Safety of Large Language Models that Follow Instructionscited by 377
  21. 2023Principle-Driven Self-Alignment of Language Models from Scratch with Minimal Human Supervisioncited by 318
  22. 2023Shadow Alignment: The Ease of Subverting Safely-Aligned Language Modelscited by 284
  23. 2022Fine-tuned Language Models are Continual Learnerscited by 152
  24. 2023GeneGPT: Augmenting Large Language Models with Domain Tools for Improved Access to Biomedical Informationcited by 115

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 18; 2025: 38; 2026: 34.

90 papers.

Sort:

By area:

  1. 2026Self-Distilled Reasoner: On-Policy Self-Distillation for Large Language Modelscited by 386
  2. 2026Reinforcement Learning via Self-Distillationcited by 285
  3. 2025CODI: Compressing Chain-of-Thought into Continuous Space via Self-Distillationcited by 216
  4. 2026Self-Distillation Enables Continual Learningcited by 212
  5. 2024Learning Dynamics of LLM Finetuningcited by 93
  6. 2024A Survey on Self-Evolution of Large Language Modelscited by 86
  7. 2025The Best Instruction-Tuning Data are Those That Fitcited by 48
  8. 2025DeSTA2.5-Audio: Toward General-Purpose Large Audio Language Model with Self-Generated Cross-Modal Alignmentcited by 47
  9. 2025The State of Multilingual LLM Safety Research: From Measuring the Language Gap to Mitigating Itcited by 42
  10. 2024LLAVADI: What Matters For Multimodal Large Language Models Distillationcited by 25
  11. 2026Aligning Language Models from User Interactionscited by 21
  12. 2025Training LLM-Based Agents with Synthetic Self-Reflected Trajectories and Partial Maskingcited by 16
38 more
  1. 2025L-MTP: Leap Multi-Token Prediction Beyond Adjacent Context for Large Language Modelscited by 14
  2. 2024E2LLM: Encoder Elongated Large Language Models for Long-Context Understanding and Reasoningcited by 13
  3. 2024Mitigating Tail Narrowing in LLM Self-Improvement via Socratic-Guided Samplingcited by 13
  4. 2025FastMTP: Accelerating LLM Inference with Enhanced Multi-Token Predictioncited by 13
  5. 2025Mitigating Forgetting in LLM Fine-Tuning via Low-Perplexity Token Learningcited by 12
  6. 2025APT: Improving Specialist LLM Performance with Weakness Case Acquisition and Iterative Preference Trainingcited by 12
  7. 2024TL-Training: A Task-Feature-Based Framework for Training Large Language Models in Tool Usecited by 11
  8. 2024Safeguard Fine-Tuned LLMs Through Pre- and Post-Tuning Model Mergingcited by 11
  9. 2025Escaping Collapse: The Strength of Weak Data for Large Language Model Trainingcited by 11
  10. 2025Krutrim LLM: Multilingual Foundational Model for over a Billion Peoplecited by 9
  11. 2025Reliable Reasoning Path: Distilling Effective Guidance for LLM Reasoning with Knowledge Graphscited by 9
  12. 2025SelfAug: Mitigating Catastrophic Forgetting in Retrieval-Augmented Generation via Distribution Self-Alignmentcited by 9
  13. 2024TAIA: Large Language Models are Out-of-Distribution Data Learnerscited by 8
  14. 2024Unified Active Retrieval for Retrieval Augmented Generationcited by 8
  15. 2024Self-Data Distillation for Recovering Quality in Pruned Large Language Modelscited by 8
  16. 2025OntoTune: Ontology-Driven Self-training for Aligning Large Language Modelscited by 8
  17. 2026Multi-Rollout On-Policy Distillation via Peer Successes and Failurescited by 8
  18. 2025Aligning Instruction Tuning with Pre-trainingcited by 7
  19. 2025Activation Approximations Can Incur Safety Vulnerabilities Even in Aligned LLMs: Comprehensive Analysis and Defensecited by 7
  20. 2025Table-r1: Self-supervised and Reinforcement Learning for Program-based Table Reasoning in Small Language Modelscited by 7
  21. 2025DIFFA: Large Language Diffusion Models Can Listen and Understandcited by 7
  22. 2025Analyzing the Effects of Supervised Fine-Tuning on Model Knowledge from Token and Parameter Levelscited by 7
  23. 2024SCAR: Data Selection via Style Consistency-Aware Response Ranking for Efficient Instruction-Tuning of Large Language Modelscited by 6
  24. 2025Flex-Judge: Text-Only Reasoning Unleashes Zero-Shot Multimodal Evaluatorscited by 6
  25. 2024Boosting LLM Translation Skills without General Ability Loss via Rationale Distillationcited by 5
  26. 2025Improving MLLM's Document Image Machine Translation via Synchronously Self-reviewing Its OCR Proficiencycited by 5
  27. 2026Large Language Model Post-Training: A Unified View of Off-Policy and On-Policy Learningcited by 5
  28. 2024Uncertainty-Guided Self-Questioning and Answering for Video-Language Alignmentcited by 4
  29. 2024Self-Powered LLM Modality Expansion for Large Speech-Text Modelscited by 4
  30. 2025MASSV: Multimodal Adaptation and Self-Data Distillation for Speculative Decoding of Vision-Language Modelscited by 4
  31. 2025InCo-DPO: Balancing Distribution Shift and Data Quality for Enhanced Preference Optimizationcited by 3
  32. 2025TAG-INSTRUCT: Controlled Instruction Complexity Enhancement through Structure-based Augmentationcited by 3
  33. 2025SWAA: Sliding Window Attention Adaptation for Efficient and Quality Preserving Long Context Processingcited by 3
  34. 2026Mitigating Safety Tax via Distribution-Grounded Refinement in Large Reasoning Modelscited by 3
  35. 2024Diabetica: Adapting Large Language Model to Enhance Multiple Medical Tasks in Diabetes Care and Managementcited by 2
  36. 2025FineScope : SAE-guided Data Selection Enables Domain Specific LLM Pruning and Finetuningcited by 2
  37. 2025Learning from the Undesirable: Robust Adaptation of Language Models without Forgettingcited by 2
  38. 2026Updating Parametric Knowledge with Context Distillation Retains Post-Training Capabilitiescited by 2

50 most cited of 90 All 90 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2026Talking to Yourself: Defying Forgetting in Large Language Models
  2. 2023Understanding Catastrophic Forgetting in Language Models via Implicit Inference
  3. 2024HFT: Half Fine-Tuning for Large Language Models
  4. 2025Catastrophic Forgetting in LLMs: A Comparative Analysis Across Language Tasks
  5. 2024Large Language Models have Intrinsic Self-Correction Ability
  6. 2025Distilled Pretraining: A modern lens of Data, In-Context Learning and Test-Time Scaling
  7. 2026Mechanistic Analysis of Catastrophic Forgetting in Large Language Models During Continual Fine-tuning
  8. 2024Efficient Knowledge Injection in LLMs via Self-Distillation