mlsanityData to 7 Oct 2026

RAIL-KD: RAndom Intermediate Layer Mapping for Knowledge Distillation

arXiv 2109.10164 ↗First posted 21 Sep 2021cs.CLcited by 18

Not on the map. Why?

Abstract (arXiv)

Intermediate layer knowledge distillation (KD) can improve the standard KD technique (which only targets the output of teacher and student models) especially over large pre-trained language models. However, intermediate layer distillation suffers from excessive computational burdens and engineering efforts required for setting up a proper layer mapping. To address these problems, we propose a RAndom Intermediate Layer Knowledge Distillation (RAIL-KD) approach in which, intermediate layers from the teacher model are selected randomly to be distilled into the intermediate layers of the student model. This randomized selection enforce that: all teacher layers are taken into account in the training process, while reducing the computational cost of intermediate layer distillation. Also, we show that it act as a regularizer for improving the generalizability of the student model. We perform extensive experiments on GLUE tasks as well as on out-of-domain test sets. We show that our proposed RAIL-KD approach outperforms other state-of-the-art intermediate layer KD methods considerably in both performance and training-time.

Key facts. Cited by 18 papers in the map (data through 7 Oct 2026). Builds on 47 of its 58 references that are in the map. First posted on arXiv 21 Sep 2021.

Builds on

47 of 58 references.

References · 47

  1. 2017Attention Is All You Needcited by 59,932
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  3. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,160
  4. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,266
  5. 2015Distilling the Knowledge in a Neural Networkcited by 10,076
  6. 2018GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understandingcited by 4,854
  7. 2019DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lightercited by 4,243
  8. 2019XLNet: Generalized Autoregressive Pretraining for Language Understandingcited by 2,801
  9. 2019SuperGLUE: A Stickier Benchmark for General-Purpose Language Understanding Systemscited by 1,718
  10. 2018Know What You Don't Know: Unanswerable Questions for SQuADcited by 1,538
  11. 2020MiniLM: Deep Self-Attention Distillation for Task-Agnostic Compression of Pre-Trained Transformerscited by 1,414
  12. 2019TinyBERT: Distilling BERT for Natural Language Understandingcited by 1,153
35 more
  1. 2017Mixed Precision Trainingcited by 1,135
  2. 2019Right for the Wrong Reasons: Diagnosing Syntactic Heuristics in Natural Language Inferencecited by 864
  3. 2019Are Sixteen Heads Really Better than One?cited by 803
  4. 2019Contrastive Representation Distillationcited by 734
  5. 2019Patient Knowledge Distillation for BERT Model Compressioncited by 502
  6. 2020MobileBERT: a Compact Task-Agnostic BERT for Resource-Limited Devicescited by 476
  7. 2019Reducing Transformer Depth on Demand with Structured Dropoutcited by 434
  8. 2019PAWS: Paraphrase Adversaries from Word Scramblingcited by 395
  9. 2019Q-BERT: Hessian Based Ultra Low Precision Quantization of BERTcited by 355
  10. 2019Q8BERT: Quantized 8Bit BERTcited by 296
  11. 2019Don't Take the Easy Way Out: Ensemble Based Methods for Avoiding Known Dataset Biasescited by 279
  12. 2020DynaBERT: Dynamic BERT with Adaptive Width and Depthcited by 246
  13. 2020MiniLMv2: Multi-Head Self-Attention Relation Distillation for Compressing Pretrained Transformerscited by 240
  14. 2020Compressing BERT: Studying the Effects of Weight Pruning on Transfer Learningcited by 184
  15. 2019Towards Debiasing Fact Verification Modelscited by 143
  16. 2019End-to-End Bias Mitigation by Modelling Biases in Corporacited by 127
  17. 2020Lite Transformer with Long-Short Range Attentioncited by 127
  18. 2020Towards Debiasing NLU Models from Unknown Biasescited by 116
  19. 2020Learning from others' mistakes: Avoiding dataset biases without modeling themcited by 83
  20. 2021Show, Attend and Distill:Knowledge Distillation via Attention-based Feature Matchingcited by 76
  21. 2020ALP-KD: Attention-Based Layer Projection for Knowledge Distillationcited by 65
  22. 2021Annealing Knowledge Distillationcited by 61
  23. 2020Contrastive Distillation on Intermediate Representations for Language Model Compressioncited by 57
  24. 2019Fully Quantized Transformer for Machine Translationcited by 36
  25. 2019Reweighted Proximal Pruning for Large-Scale Language Representationcited by 29
  26. 2021MATE-KD: Masked Adversarial TExt, a Companion to Knowledge Distillationcited by 27
  27. 2020Why Skip If You Can Combine: A Simple Knowledge Distillation Technique for Intermediate Layerscited by 24
  28. 2021End-to-End Self-Debiasing Framework for Robust NLU Trainingcited by 24
  29. 2020Towards Zero-Shot Knowledge Distillation for Natural Language Processingcited by 20
  30. 2021Context-aware Adversarial Training for Name Regularity Bias in Named Entity Recognitioncited by 19
  31. 2021KroneckerBERT: Learning Kronecker Decomposition for Pre-trained Language Models via Knowledge Distillationcited by 19
  32. 2021Not Far Away, Not So Close: Sample Efficient Nearest Neighbour Data Augmentation via MiniMaxcited by 10
  33. 2020Role-Wise Data Augmentation for Knowledge Distillationcited by 8
  34. 2021How to Select One Among All? An Extensive Empirical Study Towards the Robustness of Knowledge Distillation in Natural Language Understandingcited by 4
  35. 2021From Fully Trained to Fully Random Embeddings: Improving Neural Machine Translation with Compact Word Embedding Tablescited by 2

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 1; 2022: 5; 2023: 6; 2024: 3; 2025: 2; 2026: 1.

18 papers.

Sort:

By area:

  1. 2022In-context Learning Distillation: Transferring Few-shot Learning Ability of Pre-trained Language Modelscited by 36
  2. 2024APT: Adaptive Pruning and Tuning Pretrained Language Models for Efficient Training and Inferencecited by 29
  3. 2022Deep versus Wide: An Analysis of Student Architectures for Task-Agnostic Knowledge Distillation of Self-Supervised Speech Modelscited by 28
  4. 2021A Short Study on Compressing Decoder-Based Language Modelscited by 16
  5. 2023GKD: A General Knowledge Distillation Framework for Large-scale Pre-trained Language Modelcited by 15
  6. 2024A Comprehensive Survey of Compression Algorithms for Language Modelscited by 15
  7. 2022Multi-Level Knowledge Distillation for Out-of-Distribution Detection in Textcited by 11
  8. 2023Low-Rank Prune-And-Factorize for Language Model Compressioncited by 11
  9. 2023A Comparative Analysis of Task-Agnostic Distillation Methods for Compressing Transformer Language Modelscited by 6
  10. 2025Revisiting Intermediate-Layer Matching in Knowledge Distillation: Layer-Selection Strategy Doesn't Matter (Much)cited by 5
  11. 2023Comparative Knowledge Distillationcited by 4
  12. 2025Preventing Shortcut Learning in Medical Image Analysis through Intermediate Layer Knowledge Distillation from Specialist Teacherscited by 4
6 more
  1. 2022Continuation KD: Improved Knowledge Distillation through the Lens of Continuation Optimizationcited by 3
  2. 2024Overcoming Uncertain Incompleteness for Robust Multimodal Sequential Diagnosis Prediction via Curriculum Data Erasing Guided Knowledge Distillationcited by 3
  3. 2023Multiple Representation Transfer from Large Language Models to End-to-End ASR Systemscited by 2
  4. 2022LEAD: Liberal Feature-based Distillation for Dense Retrievalcited by 1
  5. 2023Are Intermediate Layers and Labels Really Necessary? A General Language Model Distillation Methodcited by 0
  6. 2026From Images to Words: Efficient Cross-Modal Knowledge Distillation to Language Models from Black-box Teacherscited by 0