mlsanityData to 14 Oct 2026

AD-KD: Attribution-Driven Knowledge Distillation for Language Model Compression

arXiv 2305.10010 ↗First posted 17 May 2023cs.CLcited by 26

On the map: Knowledge distillation → Knowledge distillation for LLMs

Abstract (arXiv)

Knowledge distillation has attracted a great deal of interest recently to compress pre-trained language models. However, existing knowledge distillation methods suffer from two limitations. First, the student model simply imitates the teacher's behavior while ignoring the underlying reasoning. Second, these methods usually focus on the transfer of sophisticated model-specific knowledge but overlook data-specific knowledge. In this paper, we present a novel attribution-driven knowledge distillation approach, which explores the token-level rationale behind the teacher model based on Integrated Gradients (IG) and transfers attribution knowledge to the student model. To enhance the knowledge transfer of model reasoning and generalization, we further explore multi-view attribution distillation on all potential decisions of the teacher. Comprehensive experiments are conducted with BERT on the GLUE benchmark. The experimental results demonstrate the superior performance of our approach to several state-of-the-art methods.

Key facts. Cited by 26 papers in the map (data through 14 Oct 2026). Placed in Knowledge distillation for LLMs, one of 12 directions in Knowledge distillation. Builds on 37 of its 58 references that are in the map. First posted on arXiv 17 May 2023.

Builds on

37 of 58 references.

Closest ancestors · 1

Same direction, newest first

  1. 2021Distilling Linguistic Context for Language Model Compression17 Sep

Other references · 36

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  2. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,372
  3. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,459
  4. 2015Distilling the Knowledge in a Neural Networkcited by 10,324
  5. 2018GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understandingcited by 4,899
  6. 2019DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lightercited by 4,307
  7. 2016SQuAD: 100,000+ Questions for Machine Comprehension of Textcited by 4,148
  8. 2017Axiomatic Attribution for Deep Networkscited by 3,384
  9. 2017A Broad-Coverage Challenge Corpus for Sentence Understanding through Inferencecited by 2,682
  10. 2013Visualizing and Understanding Convolutional Networkscited by 2,290
  11. 2019SuperGLUE: A Stickier Benchmark for General-Purpose Language Understanding Systemscited by 1,729
  12. 2020Knowledge Distillation: A Surveycited by 1,603
24 more
  1. 2020MiniLM: Deep Self-Attention Distillation for Task-Agnostic Compression of Pre-Trained Transformerscited by 1,463
  2. 2017Learning Important Features Through Propagating Activation Differencescited by 1,458
  3. 2019TinyBERT: Distilling BERT for Natural Language Understandingcited by 1,167
  4. 2017SemEval-2017 Task 1: Semantic Textual Similarity - Multilingual and Cross-lingual Focused Evaluationcited by 1,001
  5. 2018Neural Network Acceptability Judgmentscited by 958
  6. 2019Attention is not Explanationcited by 929
  7. 2019Are Sixteen Heads Really Better than One?cited by 816
  8. 2019Attention is not not Explanationcited by 636
  9. 2019Patient Knowledge Distillation for BERT Model Compressioncited by 508
  10. 2020MobileBERT: a Compact Task-Agnostic BERT for Resource-Limited Devicescited by 482
  11. 2020MiniLMv2: Multi-Head Self-Attention Relation Distillation for Compressing Pretrained Transformerscited by 245
  12. 2015Visualizing and Understanding Neural Models in NLPcited by 200
  13. 2020A Diagnostic Study of Explainability Techniques for Text Classificationcited by 159
  14. 2020The elephant in the interpretability room: Why use attention as explanation when we have saliency methods?cited by 122
  15. 2019On Identifiability in Transformerscited by 116
  16. 2020Restricting the Flow: Information Bottlenecks for Attributioncited by 110
  17. 2020Explaining Explanations: Axiomatic Feature Interactions for Deep Networkscited by 103
  18. 2022Less is More: Task-aware Layer-wise Distillation for Language Model Compressioncited by 93
  19. 2020How do Decisions Emerge across Layers in Neural Models? Interpretation with Differentiable Maskingcited by 63
  20. 2020Contrastive Distillation on Intermediate Representations for Language Model Compressioncited by 57
  21. 2019Saliency-driven Word Alignment Interpretation for Neural Machine Translationcited by 27
  22. 2022AdapLeR: Speeding up Inference by Adaptive Length Reductioncited by 19
  23. 2022Quantifying the Knowledge in a DNN to Explain Knowledge Distillation for Classificationcited by 19
  24. 2020Telling BERT's full story: from Local Attention to Global Aggregationcited by 15

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 3; 2024: 10; 2025: 10; 2026: 3.

26 papers.

Sort:

By area:

  1. 2024LLM Inference Unveiled: Survey and Roofline Model Insightscited by 152
  2. 2024Model Compression and Efficient Inference for Large Language Models: A Surveycited by 64
  3. 2024Survey on Knowledge Distillation for Large Language Models: Methods, Evaluation, and Applicationcited by 56
  4. 2024FuseChat: Knowledge Fusion of Chat Modelscited by 55
  5. 2025Sparse Logit Sampling: Accelerating Knowledge Distillation in LLMscited by 16
  6. 2024Sinkhorn Distance Minimization for Knowledge Distillationcited by 15
  7. 2024AMD: Automatic Multi-step Distillation of Large-scale Vision Modelscited by 12
  8. 2026OVD: On-policy Verbal Distillationcited by 11
  9. 2025Randomly Removing 50% of Dimensions in Text Embeddings has Minimal Impact on Retrieval and Classification Taskscited by 8
  10. 2025GrAInS: Gradient-based Attribution for Inference-Time Steering of LLMs and VLMscited by 6
  11. 2023rm SP3: Enhancing Structured Pruning via PCA Projectioncited by 5
  12. 2024Over-parameterized Student Model via Tensor Decomposition Boosted Knowledge Distillationcited by 5
14 more
  1. 2025Advantage-Guided Distillation for Preference Alignment in Small Language Modelscited by 5
  2. 2025DGS-Net: Distillation-Guided Gradient Surgery for CLIP Fine-Tuning in AI-Generated Image Detectioncited by 5
  3. 2024Head-wise Shareable Attention for Large Language Modelscited by 4
  4. 2024Efficient Knowledge Distillation: Empowering Small Language Models with Teacher Model Insightscited by 4
  5. 2023LumiNet: Perception-Driven Knowledge Distillation via Statistical Logit Calibrationcited by 3
  6. 2023Retrieval-based Knowledge Transfer: An Effective Approach for Extreme Large Language Model Compressioncited by 3
  7. 2025Self-Enhanced Reasoning Training: Activating Latent Reasoning in Small Models for Enhanced Reasoning Distillationcited by 3
  8. 2025Knowledge Distillation: Enhancing Neural Network Compression with Integrated Gradientscited by 2
  9. 2025An Empirical Study of Knowledge Distillation for Code Understanding Taskscited by 2
  10. 2026Trust the uncertain teacher: distilling dark knowledge via calibrated uncertaintycited by 2
  11. 2026Late-to-Early Training: LET LLMs Learn Earlier, So Faster and Bettercited by 1
  12. 2024Dynamic Self-Distillation via Previous Mini-batches for Fine-tuning Small Language Modelscited by 0
  13. 2025Model compression using knowledge distillation with integrated gradientscited by 0
  14. 2025Hybrid Attribution Priors for Explainable and Robust Model Trainingcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024Enhancing Knowledge Distillation of Large Language Models through Efficient Multi-Modal Distribution Alignment
  2. 2024MLKD-BERT: Multi-level Knowledge Distillation for Pre-trained Language Models
  3. 2023Improving Knowledge Distillation for BERT Models: Loss Functions, Mapping Methods, and Weight Tuning
  4. 2022Gradient Knowledge Distillation for Pre-trained Language Models
  5. 2023On-Policy Distillation of Language Models: Learning from Self-Generated Mistakes
  6. 2023Revisiting Intermediate Layer Distillation for Compressing Language Models: An Overfitting Perspective
  7. 2024DistiLLM: Towards Streamlined Distillation for Large Language Models
  8. 2025Every Expert Matters: Towards Effective Knowledge Distillation for Mixture-of-Experts Language Models