mlsanityData to 14 Oct 2026

ALP-KD: Attention-Based Layer Projection for Knowledge Distillation

arXiv 2012.14022 ↗First posted 27 Dec 2020cs.CLcited by 66

Not on the map. Why?

Cited by 36 in Knowledge distillation, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

Knowledge distillation is considered as a training and compression strategy in which two neural networks, namely a teacher and a student, are coupled together during training. The teacher network is supposed to be a trustworthy predictor and the student tries to mimic its predictions. Usually, a student with a lighter architecture is selected so we can achieve compression and yet deliver high-quality results. In such a setting, distillation only happens for final predictions whereas the student could also benefit from teacher's supervision for internal components. Motivated by this, we studied the problem of distillation for intermediate layers. Since there might not be a one-to-one alignment between student and teacher layers, existing techniques skip some teacher layers and only distill from a subset of them. This shortcoming directly impacts quality, so we instead propose a combinatorial technique which relies on attention. Our model fuses teacher-side information and takes each layer's significance into consideration, then performs distillation between combined teacher layers and those of the student. Using our technique, we distilled a 12-layer BERT (Devlin et al. 2019) into 6-, 4-, and 2-layer counterparts and evaluated them on GLUE tasks (Wang et al. 2018). Experimental results show that our combinatorial approach is able to outperform other existing techniques.

Key facts. Cited by 66 papers in the map (data through 14 Oct 2026). First posted on arXiv 27 Dec 2020.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2021: 11; 2022: 11; 2023: 26; 2024: 11; 2025: 4; 2026: 3.

66 papers.

Sort:

By area:

  1. 2023CAMEL: Communicative Agents for "Mind" Exploration of Large Language Model Societycited by 1,395
  2. 2022DyLoRA: Parameter Efficient Tuning of Pre-trained Models using Dynamic Search-Free Low-Rank Adaptationcited by 233
  3. 2023Fine-tuning can cripple your foundation model; preserving features may be the solutioncited by 96
  4. 2022Learning Affordance Grounding from Exocentric Imagescited by 82
  5. 2023Leveraging Vision-Language Models for Improving Domain Generalization in Image Classificationcited by 41
  6. 2023A Sentence Speaks a Thousand Images: Domain Generalization through Distilling CLIP with Language Guidancecited by 40
  7. 2021Edge-Cloud Polarization and Collaboration: A Comprehensive Survey for AIcited by 39
  8. 2022Deep versus Wide: An Analysis of Student Architectures for Task-Agnostic Knowledge Distillation of Self-Supervised Speech Modelscited by 31
  9. 2021Kronecker Decomposition for GPT Compressioncited by 28
  10. 2023Categories of Response-Based, Feature-Based, and Relation-Based Knowledge Distillationcited by 20
  11. 2021RAIL-KD: RAndom Intermediate Layer Mapping for Knowledge Distillationcited by 19
  12. 2021Pro-KD: Progressive Distillation by Following the Footsteps of the Teachercited by 17
38 more
  1. 2021A Short Study on Compressing Decoder-Based Language Modelscited by 16
  2. 2021How and When Adversarial Robustness Transfers in Knowledge Distillation?cited by 16
  3. 2023GKD: A General Knowledge Distillation Framework for Large-scale Pre-trained Language Modelcited by 15
  4. 2024A Comprehensive Survey of Compression Algorithms for Language Modelscited by 15
  5. 2024AMD: Automatic Multi-step Distillation of Large-scale Vision Modelscited by 12
  6. 2022Knowledge Distillation of Transformer-based Language Models Revisitedcited by 11
  7. 2023Student-friendly Knowledge Distillationcited by 11
  8. 2023Enhanced Multimodal Representation Learning with Cross-modal KDcited by 11
  9. 2023Cloud-Device Collaborative Learning for Multimodal Large Language Modelscited by 11
  10. 2021Not Far Away, Not So Close: Sample Efficient Nearest Neighbour Data Augmentation via MiniMaxcited by 10
  11. 2023Diverse Target and Contribution Scheduling for Domain Generalizationcited by 10
  12. 2023How to Distill your BERT: An Empirical Study on the Impact of Weight Initialisation and Distillation Objectivescited by 9
  13. 2024Real World Federated Learning with a Knowledge Distilled Transformer for Cardiac CT Imagingcited by 8
  14. 2024VLM-KD: Knowledge Distillation from VLM for Long-Tail Visual Recognitioncited by 7
  15. 2021Marginal Utility Diminishes: Exploring the Minimum Knowledge for BERT Knowledge Distillationcited by 6
  16. 2023Graph-based Knowledge Distillation: A survey and experimental evaluationcited by 6
  17. 2023A Comparative Analysis of Task-Agnostic Distillation Methods for Compressing Transformer Language Modelscited by 6
  18. 2023PreQuant: A Task-agnostic Quantization Approach for Pre-trained Language Modelscited by 5
  19. 2024Align-to-Distill: Trainable Attention Alignment for Knowledge Distillation in Neural Machine Translationcited by 5
  20. 2024Fully Exploiting Every Real Sample: SuperPixel Sample Gradient Model Stealingcited by 5
  21. 2025Revisiting Intermediate-Layer Matching in Knowledge Distillation: Layer-Selection Strategy Doesn't Matter (Much)cited by 5
  22. 2025Uncertainty-Aware Knowledge Distillation for Compact and Efficient 6DoF Pose Estimationcited by 5
  23. 2021How to Select One Among All? An Extensive Empirical Study Towards the Robustness of Knowledge Distillation in Natural Language Understandingcited by 4
  24. 2022Enabling All In-Edge Deep Learning: A Literature Reviewcited by 4
  25. 2022CILDA: Contrastive Data Augmentation using Intermediate Layer Knowledge Distillationcited by 4
  26. 2023One-stage Modality Distillation for Incomplete Multimodal Learningcited by 4
  27. 2023Data Upcycling Knowledge Distillation for Image Super-Resolutioncited by 4
  28. 2024Graph-Structured Speculative Decodingcited by 4
  29. 2022Aligning Logits Generatively for Principled Black-Box Knowledge Distillationcited by 3
  30. 2023Audio Representation Learning by Distilling Video as Privileged Informationcited by 3
  31. 2023KDSTM: Neural Semi-supervised Topic Modeling with Knowledge Distillationcited by 3
  32. 2022Kronecker Decomposition for Knowledge Graph Embeddingscited by 2
  33. 2022Do we need Label Regularization to Fine-tune Pre-trained Language Models?cited by 2
  34. 2023Heterogeneous Generative Knowledge Distillation with Masked Image Modelingcited by 2
  35. 2023What is Lost in Knowledge Distillation?cited by 2
  36. 2025Soft Knowledge Distillation with Multi-Dimensional Cross-Net Attention for Image Restoration Models Compressioncited by 2
  37. 2021Knowledge Distillation with Noisy Labels for Natural Language Understandingcited by 1
  38. 2022Selective Cross-Task Distillationcited by 1

50 most cited of 66 All 66 (JSON)