mlsanityData to 14 Oct 2026

Knowledge Distillation and Dataset Distillation of Large Language Models: Emerging Trends, Challenges, and Future Directions

arXiv 2504.14772 ↗First posted 20 Apr 2025cs.CL cs.LG stat.MLcited by 26

On the map: Knowledge distillation → Knowledge distillation for LLMs

Abstract (arXiv)

The exponential growth of Large Language Models (LLMs) continues to highlight the need for efficient strategies to meet ever-expanding computational and data demands. This survey provides a comprehensive analysis of two complementary paradigms: Knowledge Distillation (KD) and Dataset Distillation (DD), both aimed at compressing LLMs while preserving their advanced reasoning capabilities and linguistic diversity. We first examine key methodologies in KD, such as task-specific alignment, rationale-based training, and multi-teacher frameworks, alongside DD techniques that synthesize compact, high-impact datasets through optimization-based gradient matching, latent space regularization, and generative synthesis. Building on these foundations, we explore how integrating KD and DD can produce more effective and scalable compression strategies. Together, these approaches address persistent challenges in model scalability, architectural heterogeneity, and the preservation of emergent LLM abilities. We further highlight applications across domains such as healthcare and education, where distillation enables efficient deployment without sacrificing performance. Despite substantial progress, open challenges remain in preserving emergent reasoning and linguistic diversity, enabling efficient adaptation to continually evolving teacher models and datasets, and establishing comprehensive evaluation protocols. By synthesizing methodological innovations, theoretical foundations, and practical insights, our survey charts a path toward sustainable, resource-efficient LLMs through the tighter integration of KD and DD principles.

Key facts. Cited by 26 papers in the map (data through 14 Oct 2026). Placed in Knowledge distillation for LLMs, one of 12 directions in Knowledge distillation. Builds on 37 of its 63 references that are in the map. First posted on arXiv 20 Apr 2025.

Builds on

37 of 63 references.

Closest ancestors · 3

Same direction, newest first

  1. 2025Who Taught You That? Tracing Teachers in Model Distillation10 Feb
  2. 2024Beyond Answers: Transferring Reasoning Capabilities to Smaller LLMs Using Multi-Teacher Knowledge Distillation7 Feb
  3. 2021Generate, Annotate, and Learn: NLP with Synthetic Text11 Jun

Other references · 34

  1. 2016Overcoming catastrophic forgetting in neural networkscited by 4,787
  2. 2022Self-Instruct: Aligning Language Models with Self-Generated Instructionscited by 2,724
  3. 2020Knowledge Distillation: A Surveycited by 1,603
  4. 2023A Survey on Hallucination in Large Language Models: Principles, Taxonomy, Challenges, and Open Questionscited by 1,601
  5. 2020Don't Stop Pretraining: Adapt Language Models to Domains and Taskscited by 1,565
  6. 2016Variational Inference: A Review for Statisticianscited by 1,472
  7. 2023FActScore: Fine-grained Atomic Evaluation of Factual Precision in Long Form Text Generationcited by 1,282
  8. 2023A Survey on Evaluation of Large Language Modelscited by 1,242
  9. 2019TinyBERT: Distilling BERT for Natural Language Understandingcited by 1,167
  10. 2023Just Ask for Calibration: Strategies for Eliciting Calibrated Confidence Scores from Language Models Fine-Tuned with Human Feedbackcited by 812
  11. 2020Efficient Transformers: A Surveycited by 790
  12. 2023Distilling Step-by-Step! Outperforming Larger Language Models with Less Training Data and Smaller Model Sizescited by 737
22 more
  1. 2023Automatic Prompt Optimization with "Gradient Descent" and Beam Searchcited by 653
  2. 2022RLPrompt: Optimizing Discrete Text Prompts with Reinforcement Learningcited by 366
  3. 2023PromptRobust: Towards Evaluating the Robustness of Large Language Models on Adversarial Promptscited by 226
  4. 2024A Comprehensive Survey of Small Language Models in the Era of Large Language Models: Techniques, Enhancements, Applications, Collaboration with LLMs, and Trustworthinesscited by 127
  5. 2023Factuality Challenges in the Era of Large Language Modelscited by 126
  6. 2023LaMini-LM: A Diverse Herd of Distilled Models from Large-Scale Instructionscited by 124
  7. 2023Synthetic Data Generation with Large Language Models for Text Classification: Potential and Limitationscited by 100
  8. 2024Safeguarding Large Language Models: A Surveycited by 78
  9. 2024FedBiOT: LLM Local Fine-tuning in Federated Learning without Full Modelcited by 74
  10. 2022PromDA: Prompt-based Data Augmentation for Low-Resource NLU Taskscited by 51
  11. 2024TAGCOS: Task-agnostic Gradient Clustered Coreset Selection for Instruction Tuning Datacited by 30
  12. 2024DiLM: Distilling Dataset into Language Model for Text-level Dataset Distillationcited by 20
  13. 2023Distilling Large Language Models for Matching Patients to Clinical Trialscited by 16
  14. 2024FIRST: Teach A Reliable Large Language Model Through Efficient Trustworthy Distillationcited by 11
  15. 2024SEEKR: Selective Attention-Guided Knowledge Retention for Continual Learning of Large Language Modelscited by 11
  16. 2024DiffLM: Controllable Synthetic Data Generation via Diffusion Language Modelscited by 9
  17. 2025Quantification of Large Language Model Distillationcited by 9
  18. 2020More efficient approximation of smoothing splines via space-filling basis selectioncited by 7
  19. 2024Noisy Node Classification by Bi-level Optimization based Multi-teacher Distillationcited by 4
  20. 2024LLM Distillation for Efficient Few-Shot Multiple Choice Question Answeringcited by 4
  21. 2022Core-Elements for Large-Scale Least Squares Estimationcited by 3
  22. 2024ChuLo: Chunk-Level Key Information Representation for Long Document Understandingcited by 2

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 11; 2026: 15.

26 papers.

Sort:

By area:

  1. 2025Alignment and Safety in Large Language Models: Safety Mechanisms, Training Paradigms, and Emerging Challengescited by 15
  2. 2026Zero Sum SVD: Balancing Loss Sensitivity for Low Rank LLM Compressioncited by 8
  3. 2025Learning the Wrong Lessons: Syntactic-Domain Spurious Correlations in Language Modelscited by 6
  4. 2025Reasoning Scaffolding: Distilling the Flow of Thought from LLMscited by 5
  5. 2025Few-Shot Knowledge Distillation of LLMs With Counterfactual Explanationscited by 4
  6. 2025Knowledge-Driven Agentic Scientific Corpus Distillation Framework for Biomedical Large Language Models Trainingcited by 1
  7. 2025Domain-Adaptive Diagnosis of Lewy Body Disease with Transferability Aware Transformercited by 1
  8. 2025Context-Adaptive Synthesis and Compression for Enhanced Retrieval-Augmented Generation in Complex Domainscited by 1
  9. 2025CKD-EHR:Clinical Knowledge Distillation for Electronic Health Recordscited by 0
  10. 2025Efficient Graph Knowledge Distillation from GNNs to Kolmogorov--Arnold Networks via Self-Attention Dynamic Samplingcited by 0
  11. 2025Data Trajectory Alignment for LLM Domain Adaptation: A Two-Phase Synthesis Framework for Telecommunications Mathematicscited by 0
  12. 2025Fast Reasoning Segmentation for Images and Videoscited by 0
14 more
  1. 2026Domain Specific Specialization in Low-Resource Settings: The Efficacy of Offline Response-Based Knowledge Distillation in Large Language Modelscited by 0
  2. 2026Gradually Compacting Large Language Models for Reasoning Like a Boiling Frogcited by 0
  3. 2026Efficient endometrial carcinoma screening via cross-modal synthesis and gradient distillationcited by 0
  4. 2026DAIT: Distillation from Vision-Language Models to Lightweight Classifiers with Adaptive Intermediate Teacher Transfercited by 0
  5. 2026Big2Small: A Unifying Neural Network Framework for Model Compressioncited by 0
  6. 2026Omnimodal Dataset Distillation via High-order Proxy Alignmentcited by 0
  7. 2026Evolving Knowledge Distillation for Lightweight Neural Machine Translationcited by 0
  8. 2026Characterize Then Distill: Mechanistic Reasoning in Large Output Spacescited by 0
  9. 2026DynaOD: Dynamic Origin-Destination Flow Generation with Discrete-to-Continuous Temporal Semantic Modelingcited by 0
  10. 2026EPTS: Elastic Post-Training Sparsity for Efficient Large Language Model Compressioncited by 0
  11. 2026Knowledge Cascade: Reverse Knowledge Distillation on Nonparametric Multivariate Functional Estimationcited by 0
  12. 2026A Statistical Perspective on Knowledge Distillation: Foundations, Classical Methods, and Large Language Model Extensionscited by 0
  13. 2026Revisiting On-policy Adversarial Black-Box Distillation: Calibrating Groupwise Reward Geometry for Effective Advantage Constructioncited by 0
  14. 2026COFFEQA: Counterfactual-Guided Few-Shot Distillation for LLM Question Answeringcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024Survey on Knowledge Distillation for Large Language Models: Methods, Evaluation, and Application
  2. 2024A Survey on Knowledge Distillation of Large Language Models
  3. 2024DDK: Distilling Domain Knowledge for Efficient Large Language Models
  4. 2024DistiLLM: Towards Streamlined Distillation for Large Language Models
  5. 2024Multi-Stage Balanced Distillation: Addressing Long-Tail Challenges in Sequence-Level Knowledge Distillation
  6. 2025Less is More: Selective Reflection for Compatible and Efficient Knowledge Distillation in Large Language Models
  7. 2026KDFP: A first-principles approach to knowledge distillation in large language models
  8. 2024Dual-Space Knowledge Distillation for Large Language Models