mlsanityData to 14 Oct 2026

Minifinetuning: Low-Data Generation Domain Adaptation through Corrective Self-Distillation

arXiv 2506.15702 ↗First posted 30 May 2025cs.LG cs.AIcited by 1

On the map: Knowledge distillation → Knowledge distillation for LLMs

Abstract (arXiv)

Finetuning language models for a new domain inevitably leads to the deterioration of their general performance. This becomes more pronounced the more limited the finetuning data resource. We introduce minifinetuning (MFT), a method for language model domain adaptation that considerably reduces the effects of overfitting-induced degeneralization in low-data settings and which does so in the absence of any pre-training data for replay. MFT demonstrates 2-10x more favourable specialization-to-degeneralization ratios than standard finetuning across a wide range of models and domains and exhibits an intrinsic robustness to overfitting when data in the new domain is scarce and down to as little as 500 samples. Employing corrective self-distillation that is individualized on the sample level, MFT outperforms parameter-efficient finetuning methods, demonstrates replay-like degeneralization mitigation properties, and is composable with either for a combined effect.

Key facts. Cited by 1 paper in the map (data through 14 Oct 2026). Placed in Knowledge distillation for LLMs, one of 12 directions in Knowledge distillation. Builds on 26 of its 36 references that are in the map. First posted on arXiv 30 May 2025.

Builds on

26 of 36 references.

Closest ancestors · 2

Same direction, newest first

  1. 2024Compact Language Models via Pruning and Knowledge Distillation19 Jul
  2. 2024Self-Distillation Bridges Distribution Gap in Language Model Fine-Tuning21 Feb

Other references · 24

  1. 2024The Llama 3 Herd of Modelscited by 14,715
  2. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,380
  3. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 13,028
  4. 2016Overcoming catastrophic forgetting in neural networkscited by 4,787
  5. 2019Parameter-Efficient Transfer Learning for NLPcited by 4,236
  6. 2024Phi-3 Technical Report: A Highly Capable Language Model Locally on Your Phonecited by 1,917
  7. 2019ZeRO: Memory Optimizations Toward Training Trillion Parameter Modelscited by 1,301
  8. 2022Few-Shot Parameter-Efficient Fine-Tuning is Better and Cheaper than In-Context Learningcited by 896
  9. 2024DoRA: Weight-Decomposed Low-Rank Adaptationcited by 758
  10. 2022Fine-Tuning can Distort Pretrained Features and Underperform Out-of-Distributioncited by 739
  11. 2023Textbooks Are All You Needcited by 569
  12. 2024Continual Learning of Large Language Models: A Comprehensive Surveycited by 234
12 more
  1. 2024When Scaling Meets LLM Finetuning: The Effect of Data, Model and Finetuning Methodcited by 204
  2. 2023OpenWebMath: An Open Dataset of High-Quality Mathematical Web Textcited by 155
  3. 2022Pile of Law: Learning Responsible Data Filtering from the Law and a 256GB Open-Source Legal Datasetcited by 113
  4. 2024Reuse, Don't Retrain: A Recipe for Continued Pretraining of Language Modelscited by 72
  5. 2023Maybe Only 0.5% Data is Needed: A Preliminary Exploration of Low Training Data Instruction Tuningcited by 58
  6. 2024Fine-tuning and Utilization Methods of Domain-specific LLMscited by 36
  7. 2021Efficient Hierarchical Domain Adaptation for Pretrained Language Modelscited by 32
  8. 2024Scalable Language Model with Generalized Continual Learningcited by 28
  9. 2020Unsupervised Domain Adaptation of a Pretrained Cross-Lingual Language Modelcited by 24
  10. 2023kNN-Adapter: Efficient Domain Adaptation for Black-Box Language Modelscited by 14
  11. 2020Effective Unsupervised Domain Adaptation with Adversarially Trained Language Modelscited by 12
  12. 2024Unlocking Parameter-Efficient Fine-Tuning for Low-Resource Language Translationcited by 4

Built on by

Cited by year

Citing papers per year: 2025: 1.

1 paper.

  1. 2025Small Language Models are the Future of Agentic AIcited by 270

Similar, not cited

Close in wording, no citation either way

  1. 2024Fine-tuning large language models for domain adaptation: Exploration of training strategies, scaling, model merging and synergistic capabilities
  2. 2024Self-Data Distillation for Recovering Quality in Pruned Large Language Models
  3. 2026The Finetuner's Fallacy: When to Pretrain with Your Finetuning Data
  4. 2026Domain-Specific Data Synthesis for LLMs via Minimal Sufficient Representation Learning
  5. 2026Domain Specific Specialization in Low-Resource Settings: The Efficacy of Offline Response-Based Knowledge Distillation in Large Language Models
  6. 2024RE-Adapt: Reverse Engineered Adaptation of Large Language Models
  7. 2025FineScope : SAE-guided Data Selection Enables Domain Specific LLM Pruning and Finetuning
  8. 2025Mining Useful General Data for Low-Resource Domain Adaptation