mlsanityData to 14 Oct 2026

Knowledge Distillation from Internal Representations

arXiv 1910.03723 ↗First posted 8 Oct 2019cs.CLcited by 87

Not on the map. Why?

Cited by 33 in Knowledge distillation, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

Knowledge distillation is typically conducted by training a small model (the student) to mimic a large and cumbersome model (the teacher). The idea is to compress the knowledge from the teacher by using its output probabilities as soft-labels to optimize the student. However, when the teacher is considerably large, there is no guarantee that the internal knowledge of the teacher will be transferred into the student; even if the student closely matches the soft-labels, its internal representations may be considerably different. This internal mismatch can undermine the generalization capabilities originally intended to be transferred from the teacher to the student. In this paper, we propose to distill the internal representations of a large model such as BERT into a simplified version of it. We formulate two ways to distill such representations and various algorithms to conduct the distillation. We experiment with datasets from the GLUE benchmark and consistently show that adding knowledge distillation from internal representations is a more powerful method than only using soft-label distillation.

Key facts. Cited by 87 papers in the map (data through 14 Oct 2026). First posted on arXiv 8 Oct 2019.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2021: 16; 2022: 19; 2023: 19; 2024: 16; 2025: 11; 2026: 6.

87 papers.

Sort:

By area:

  1. 2021Zen-NAS: A Zero-Shot NAS for High-Performance Deep Image Recognitioncited by 73
  2. 2021A Survey on Green Deep Learningcited by 52
  3. 2022BiT: Robustly Binarized Multi-distilled Transformercited by 52
  4. 2022LightHuBERT: Lightweight and Configurable Speech Representation Learning with Once-for-All Hidden-Unit BERTcited by 50
  5. 2022A Survey on Model Compression and Acceleration for Pretrained Language Modelscited by 45
  6. 2021There is More than Meets the Eye: Self-Supervised Multi-Object Detection and Tracking with Sound by Distilling Multimodal Knowledgecited by 44
  7. 2022Transfer without Forgettingcited by 44
  8. 2021Not All Negatives are Equal: Label-Aware Contrastive Loss for Fine-grained Text Classificationcited by 43
  9. 2022Cross-Architecture Knowledge Distillationcited by 41
  10. 2023Teacher-Student Architecture for Knowledge Distillation: A Surveycited by 26
  11. 2023HomoDistil: Homotopic Task-Agnostic Distillation of Pre-trained Transformerscited by 25
  12. 2022Teacher-Student Architecture for Knowledge Learning: A Surveycited by 23
38 more
  1. 2022Egeria: Efficient DNN Training with Knowledge-Guided Layer Freezingcited by 22
  2. 2022Mokey: Enabling Narrow Fixed-Point Inference for Out-of-the-Box Floating-Point Transformer Modelscited by 19
  3. 2022Extreme Compression for Pre-trained Transformers Made Simple and Efficientcited by 19
  4. 2023Coaching a Teachable Studentcited by 19
  5. 2021XtremeDistilTransformers: Task Transfer for Task-agnostic Distillationcited by 18
  6. 2023Rethinking Momentum Knowledge Distillation in Online Continual Learningcited by 18
  7. 2023Distilling Efficient Vision Transformers from CNNs for Semantic Segmentationcited by 18
  8. 2023GKD: A General Knowledge Distillation Framework for Large-scale Pre-trained Language Modelcited by 15
  9. 2024A Comprehensive Survey of Compression Algorithms for Language Modelscited by 15
  10. 2025Adaptive Semantic Token Communication for Transformer-based Edge Inferencecited by 14
  11. 2026Memory-Efficient Looped Transformer: Decoupling Compute from Memory in Looped Language Modelscited by 14
  12. 2022Layer Freezing & Data Sieving: Missing Pieces of a Generic Framework for Sparse Trainingcited by 11
  13. 2022Understanding and Improving Knowledge Distillation for Quantization-Aware Training of Large Transformer Encoderscited by 11
  14. 2023EmbedDistill: A Geometric Knowledge Distillation for Information Retrievalcited by 11
  15. 2023Accelerating Molecular Graph Neural Networks via Knowledge Distillationcited by 11
  16. 2024Adaptive Semantic Token Selection for AI-native Goal-oriented Communicationscited by 10
  17. 2022Cross-lingual Lifelong Learningcited by 9
  18. 2025Data Duplication: A Novel Multi-Purpose Attack Paradigm in Machine Unlearningcited by 9
  19. 2022Knowledge Distillation of Russian Language Models with Reduction of Vocabularycited by 7
  20. 2024CKD: Contrastive Knowledge Distillation from A Sample-wise Perspectivecited by 7
  21. 2021Model Compression for Domain Adaptation through Causal Effect Estimationcited by 6
  22. 2023Efficient Self-supervised Continual Learning with Progressive Task-correlated Layer Freezingcited by 6
  23. 2023Adaptive Computation Modules: Granular Conditional Computation For Efficient Inferencecited by 6
  24. 2024Factual Dialogue Summarization via Learning from Large Language Modelscited by 6
  25. 2024Leveraging Foundation Models via Knowledge Distillation in Multi-Object Tracking: Distilling DINOv2 Features to FairMOTcited by 6
  26. 2024Sorbet: A Neuromorphic Hardware-Compatible Transformer-Based Spiking Language Modelcited by 6
  27. 2021Representation Consolidation for Training Expert Studentscited by 5
  28. 2022Decentralized Learning with Multi-Headed Distillationcited by 5
  29. 2023ChromaDistill: Colorizing Monochrome Radiance Fields with Knowledge Distillationcited by 5
  30. 2024TSAK: Two-Stage Semantic-Aware Knowledge Distillation for Efficient Wearable Modality and Model Optimization in Manufacturing Linescited by 5
  31. 2021A Flexible Multi-Task Model for BERT Servingcited by 4
  32. 2024Exploring Continual Learning of Compositional Generalization in NLIcited by 4
  33. 2021Industry Scale Semi-Supervised Learning for Natural Language Understandingcited by 3
  34. 2022Aligning Logits Generatively for Principled Black-Box Knowledge Distillationcited by 3
  35. 2023Knowledge Distillation ≈ Label Smoothing: Fact or Fallacy?cited by 3
  36. 2024VTrans: Accelerating Transformer Compression with Variational Information Bottleneck based Pruningcited by 3
  37. 2026Visual-Noise Guided In-Context Distillation for Multimodal Large Language Model Unlearningcited by 3
  38. 2021Student-Teacher Learning from Clean Inputs to Noisy Inputscited by 2

50 most cited of 87 All 87 (JSON)