mlsanityData to 14 Oct 2026

Feature Alignment and Representation Transfer in Knowledge Distillation for Large Language Models

arXiv 2504.13825 ↗First posted 18 Apr 2025cs.CL cs.LGcited by 6

On the map: Knowledge distillation → General knowledge distillation methods

Abstract (arXiv)

Knowledge distillation (KD) is a technique for transferring knowledge from complex teacher models to simpler student models, significantly enhancing model efficiency and accuracy. It has demonstrated substantial advancements in various applications including image classification, object detection, language modeling, text classification, and sentiment analysis. Recent innovations in KD methods, such as attention-based approaches, block-wise logit distillation, and decoupling distillation, have notably improved student model performance. These techniques focus on stimulus complexity, attention mechanisms, and global information capture to optimize knowledge transfer. In addition, KD has proven effective in compressing large language models while preserving accuracy, reducing computational overhead, and improving inference speed. This survey synthesizes the latest literature, highlighting key findings, contributions, and future directions in knowledge distillation to provide insights for researchers and practitioners on its evolving role in artificial intelligence and machine learning.

Key facts. Cited by 6 papers in the map (data through 14 Oct 2026). Placed in General knowledge distillation methods, one of 12 directions in Knowledge distillation. Builds on 103 of its 111 references that are in the map. First posted on arXiv 18 Apr 2025.

Builds on

103 of 111 references.

Closest ancestors · 41

Same direction, newest first

  1. 2024Neural Collapse Inspired Knowledge Distillation16 Dec
  2. 2024Decoupling Dark Knowledge via Block-wise Logit Distillation for Feature-level Alignment3 Nov
  3. 2024CAKD: A Correlation-Aware Knowledge Distillation Framework Based on Decoupling Kullback-Leibler Divergence17 Oct
  4. 2024Towards Optimal Trade-offs in Knowledge Distillation for CNNs and Vision Transformers at the Edge25 Jun
  5. 2024A Comprehensive Review of Knowledge Distillation in Computer Vision1 Apr
  6. 2024Frequency Attention for Knowledge Distillation9 Mar
  7. 2023What is Lost in Knowledge Distillation?7 Nov
  8. 2023LumiNet: Perception-Driven Knowledge Distillation via Statistical Logit Calibration5 Oct
  9. 2023Can a student Large Language Model perform as well as it's teacher?3 Oct
  10. 2023Multi-Label Knowledge Distillation12 Aug
  11. 2023NormKD: Normalized Logits for Knowledge Distillation1 Aug
  12. 2023Cluster-aware Semi-supervised Learning: Relational Knowledge Distillation Provably Learns Clustering20 Jul
29 more
  1. 2023DOT: A Distillation-Oriented Trainer17 Jul
  2. 2023Categories of Response-Based, Feature-Based, and Relation-Based Knowledge Distillation19 Jun
  3. 2023Knowledge Distillation Performs Partial Variance Reduction27 May
  4. 2023Improving Knowledge Distillation via Regularizing Feature Norm and Direction26 May
  5. 2023Student-friendly Knowledge Distillation18 May
  6. 2023A Survey on Recent Teacher-student Learning Studies10 Apr
  7. 2023DisWOT: Student Architecture Search for Distillation WithOut Training28 Mar
  8. 2023Knowledge Distillation from Single to Multi Labels: an Empirical Study15 Mar
  9. 2023Knowledge Distillation for Efficient Sequences of Training Runs11 Mar
  10. 2023Explicit and Implicit Knowledge Distillation via Unlabeled Data17 Feb
  11. 2023Knowledge Distillation ≈ Label Smoothing: Fact or Fallacy?30 Jan
  12. 2022Class-aware Information for Logit-based Knowledge Distillation27 Nov
  13. 2022Quantifying the Knowledge in a DNN to Explain Knowledge Distillation for Classification18 Aug
  14. 2022Black-box Few-shot Knowledge Distillation25 Jul
  15. 2022Revisiting Label Smoothing and Knowledge Distillation Compatibility: What was Missing?29 Jun
  16. 2022InDistill: Information flow-preserving knowledge distillation for model compression20 May
  17. 2022Spot-adaptive Knowledge Distillation5 May
  18. 2022A Closer Look at Knowledge Distillation with Features, Logits, and Gradients18 Mar
  19. 2022Decoupled Knowledge Distillation16 Mar
  20. 2022Dynamic Rectification Knowledge Distillation27 Jan
  21. 2021Controlling the Quality of Distillation in Response-Based Network Compression19 Dec
  22. 2021Pixel Distillation: A New Knowledge Distillation Scheme for Low-Resolution Image Recognition17 Dec
  23. 2021Adaptive Distillation: Aggregating Knowledge from Multiple Paths for Efficient Distillation19 Oct
  24. 2021Partial to Whole Knowledge Distillation: Progressive Distilling Decomposed Knowledge Boosts Student Better26 Sep
  25. 2021Not All Knowledge Is Created Equal: Mutual Distillation of Confident Knowledge2 Jun
  26. 2021Knowledge Distillation as Semiparametric Inference20 Apr
  27. 2021A New Training Framework for Deep Neural Network12 Mar
  28. 2021Enhancing Data-Free Adversarial Distillation with Activation Regularization and Virtual Interpolation23 Feb
  29. 2021Show, Attend and Distill:Knowledge Distillation via Attention-based Feature Matching5 Feb

Other references · 62

  1. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,599
  2. 2015Distilling the Knowledge in a Neural Networkcited by 10,324
  3. 2020YOLOv4: Optimal Speed and Accuracy of Object Detectioncited by 1,630
  4. 2020Knowledge Distillation: A Surveycited by 1,603
  5. 2019When Does Label Smoothing Help?cited by 988
  6. 2020Knowledge Distillation and Student-Teacher Learning for Visual Intelligence: A Review and New Outlookscited by 316
  7. 2023f-Divergence Minimization for Sequence-Level Knowledge Distillationcited by 114
  8. 2020Understanding and Improving Knowledge Distillationcited by 83
  9. 2024Rethinking Kullback-Leibler Divergence in Knowledge Distillation for Large Language Modelscited by 65
  10. 2024Towards Cross-Tokenizer Distillation: the Universal Logit Distillation Loss for LLMscited by 63
  11. 2019Self-Knowledge Distillation in Natural Language Processingcited by 60
  12. 2024Survey on Knowledge Distillation for Large Language Models: Methods, Evaluation, and Applicationcited by 56
38 more
  1. 2021Localization Distillation for Dense Object Detectioncited by 55
  2. 2021Knowledge Distillation for Object Detection via Rank Mimicking and Prediction-guided Feature Imitationcited by 40
  3. 2021Distilling Linguistic Context for Language Model Compressioncited by 33
  4. 2020Knowledge Distillation in Deep Learning and its Applicationscited by 29
  5. 2020Prime-Aware Adaptive Distillationcited by 28
  6. 2021G-DetKD: Towards General Distillation Framework for Object Detectors via Contrastive and Semantic-guided Feature Imitationcited by 28
  7. 2024Pre-training Distillation for Large Language Models: A Design Space Explorationcited by 27
  8. 2020TextBrewer: An Open-Source Knowledge Distillation Toolkit for Natural Language Processingcited by 26
  9. 2021Data Distillation for Text Classificationcited by 25
  10. 2023HomoDistil: Homotopic Task-Agnostic Distillation of Pre-trained Transformerscited by 25
  11. 2020Weight Distillation: Transferring the Knowledge in Neural Network Parameterscited by 23
  12. 2020Channel Distillation: Channel-Wise Attention for Knowledge Distillationcited by 21
  13. 2023Is Synthetic Data From Diffusion Models Ready for Knowledge Distillation?cited by 21
  14. 2020Towards Zero-Shot Knowledge Distillation for Natural Language Processingcited by 20
  15. 2021Causal Distillation for Language Modelscited by 19
  16. 2021XtremeDistilTransformers: Task Transfer for Task-agnostic Distillationcited by 18
  17. 2022Prediction-Guided Distillation for Dense Object Detectioncited by 14
  18. 2022FAKD: Feature Augmented Knowledge Distillation for Semantic Segmentationcited by 14
  19. 2023Improved Knowledge Distillation for Pre-trained Language Models via Knowledge Selectioncited by 12
  20. 2024Sentence-Level or Token-Level? A Comprehensive Study on Knowledge Distillationcited by 12
  21. 2019Modeling Teacher-Student Techniques in Deep Neural Networks for Knowledge Distillationcited by 11
  22. 2024Optimizing YOLOv5s Object Detection through Knowledge Distillation algorithmcited by 11
  23. 2023Learning Lightweight Object Detectors via Multi-Teacher Progressive Distillationcited by 10
  24. 2019The State of Knowledge Distillation for Classificationcited by 9
  25. 2020Triplet Loss for Knowledge Distillationcited by 9
  26. 2023How to Distill your BERT: An Empirical Study on the Impact of Weight Initialisation and Distillation Objectivescited by 9
  27. 2019QUEST: Quantized embedding space for transferring knowledgecited by 7
  28. 2021Enhance Long Text Understanding via Distilled Gist Detector from Abstractive Summarizationcited by 6
  29. 2024Leveraging Large Language Models for Enhanced NLP Task Performance through Knowledge Distillation and Optimized Training Strategiescited by 6
  30. 2020Distilling Neural Networks for Greener and Faster Dependency Parsingcited by 5
  31. 2022Multi-Granularity Distillation Scheme Towards Lightweight Semi-Supervised Semantic Segmentationcited by 5
  32. 2022Gradient Knowledge Distillation for Pre-trained Language Modelscited by 5
  33. 2024Multi-Stage Balanced Distillation: Addressing Long-Tail Challenges in Sequence-Level Knowledge Distillationcited by 5
  34. 2024Make a Strong Teacher with Label Assistance: A Novel Knowledge Distillation Approach for Semantic Segmentationcited by 5
  35. 2017Knowledge distillation using unlabeled mismatched imagescited by 4
  36. 2022Adaptive Instance Distillation for Object Detection in Autonomous Drivingcited by 4
  37. 2024Mixture cure semiparametric additive hazard models under partly interval censoring -- a penalized likelihood approachcited by 4
  38. 2024The Privileged Students: On the Value of Initialization in Multilingual Knowledge Distillationcited by 4

50 most cited of 62 All 62 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 3; 2026: 3.

6 papers.

  1. 2025AutoChemSchematic AI: Agentic Physics-Aware Automation for Chemical Manufacturing Scale-Upcited by 6
  2. 2025Order-Level Attention Similarity Across Language Models: A Latent Commonalitycited by 2
  3. 2026ECR: Manifold-Guided Semantic Cues for Compact Language Modelscited by 1
  4. 2025Fast Reasoning Segmentation for Images and Videoscited by 0
  5. 2026FedCausal-Dyn: A Causal-Dynamic Paradigm for Federated Learning under Dynamic Feature Driftcited by 0
  6. 2026APQF: Agentic Profiling-Guided Structured Pruning and Mixed-Precision Quantization with Adaptive Fine-Tuningcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2021Dynamic Knowledge Distillation for Pre-trained Language Models
  2. 2024A Survey on Knowledge Distillation of Large Language Models
  3. 2024Harmonizing knowledge Transfer in Neural Network with Unified Distillation
  4. 2026Memorization Dynamics in Knowledge Distillation for Language Models
  5. 2024Dual-Space Knowledge Distillation for Large Language Models
  6. 2024DDK: Distilling Domain Knowledge for Efficient Large Language Models
  7. 2022On effects of Knowledge Distillation on Transfer Learning
  8. 2025Knowledge Distillation and Dataset Distillation of Large Language Models: Emerging Trends, Challenges, and Future Directions