mlsanityData to 13 Oct 2026

Revisiting Intermediate Layer Distillation for Compressing Language Models: An Overfitting Perspective

arXiv 2302.01530 ↗First posted 3 Feb 2023cs.CL cs.AI cs.LGcited by 5

On the map: Knowledge distillation → Knowledge distillation for LLMs

Abstract (arXiv)

Knowledge distillation (KD) is a highly promising method for mitigating the computational problems of pre-trained language models (PLMs). Among various KD approaches, Intermediate Layer Distillation (ILD) has been a de facto standard KD method with its performance efficacy in the NLP field. In this paper, we find that existing ILD methods are prone to overfitting to training datasets, although these methods transfer more information than the original KD. Next, we present the simple observations to mitigate the overfitting of ILD: distilling only the last Transformer layer and conducting ILD on supplementary tasks. Based on our two findings, we propose a simple yet effective consistency-regularized ILD (CR-ILD), which prevents the student model from overfitting the training dataset. Substantial experiments on distilling BERT on the GLUE benchmark and several synthetic datasets demonstrate that our proposed ILD method outperforms other KD techniques. Our code is available at https://github.com/jongwooko/CR-ILD.

Key facts. Cited by 5 papers in the map (data through 13 Oct 2026). Placed in Knowledge distillation for LLMs, one of 12 directions in Knowledge distillation. Builds on 30 of its 44 references that are in the map. First posted on arXiv 3 Feb 2023.

Builds on

30 of 44 references.

Closest ancestors · 2

Same direction, newest first

  1. 2021Distilling Linguistic Context for Language Model Compression17 Sep
  2. 2021MATE-KD: Masked Adversarial TExt, a Companion to Knowledge Distillation12 May

Other references · 28

  1. 2017Attention Is All You Needcited by 60,106
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,304
  3. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,549
  4. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,322
  5. 2015Distilling the Knowledge in a Neural Networkcited by 10,180
  6. 2017mixup: Beyond Empirical Risk Minimizationcited by 5,170
  7. 2018GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understandingcited by 4,880
  8. 2020FixMatch: Simplifying Semi-Supervised Learning with Consistency and Confidencecited by 2,082
  9. 2020MiniLM: Deep Self-Attention Distillation for Task-Agnostic Compression of Pre-Trained Transformerscited by 1,435
  10. 2016On Large-Batch Training for Deep Learning: Generalization Gap and Sharp Minimacited by 1,344
  11. 2019TinyBERT: Distilling BERT for Natural Language Understandingcited by 1,157
  12. 2019Patient Knowledge Distillation for BERT Model Compressioncited by 504
16 more
  1. 2019Be Your Own Teacher: Improve the Performance of Convolutional Neural Networks via Self Distillationcited by 493
  2. 2020MobileBERT: a Compact Task-Agnostic BERT for Resource-Limited Devicescited by 480
  3. 2020MiniLMv2: Multi-Head Self-Attention Relation Distillation for Compressing Pretrained Transformerscited by 240
  4. 2020Revisiting Few-sample BERT Fine-tuningcited by 237
  5. 2018Sentence Encoders on STILTs: Supplementary Training on Intermediate Labeled-data Taskscited by 214
  6. 2020MixText: Linguistically-Informed Interpolation of Hidden Space for Semi-Supervised Text Classificationcited by 186
  7. 2020BinaryBERT: Pushing the Limit of BERT Quantizationcited by 177
  8. 2020Better Fine-Tuning by Reducing Representational Collapsecited by 157
  9. 2020Intermediate-Task Transfer Learning with Pretrained Models for Natural Language Understanding: When and Why Does It Work?cited by 127
  10. 2020Consistency Regularization for Certified Robustness of Smoothed Classifierscited by 73
  11. 2020MixKD: Towards Efficient Distillation of Large-scale Language Modelscited by 65
  12. 2021Annealing Knowledge Distillationcited by 61
  13. 2020LadaBERT: Lightweight Adaptation of BERT through Hybrid Model Compressioncited by 45
  14. 2021Consistency Regularization for Cross-Lingual Fine-Tuningcited by 41
  15. 2020BERT-EMD: Many-to-Many Layer Mapping for BERT Compression with Earth Mover's Distancecited by 27
  16. 2020Why Skip If You Can Combine: A Simple Knowledge Distillation Technique for Intermediate Layerscited by 24

Built on by

Cited by year

Citing papers per year: 2023: 3; 2024: 0; 2025: 2.

5 papers.

  1. 2023Fast and Robust Early-Exiting Framework for Autoregressive Language Models with Synchronized Parallel Decodingcited by 80
  2. 2025DistiLLM-2: A Contrastive Approach Boosts the Distillation of LLMscited by 76
  3. 2023Neural Architecture Search for Effective Teacher-Student Knowledge Transfer in Language Modelscited by 6
  4. 2023A Comparative Analysis of Task-Agnostic Distillation Methods for Compressing Transformer Language Modelscited by 6
  5. 2025Revisiting Intermediate-Layer Matching in Knowledge Distillation: Layer-Selection Strategy Doesn't Matter (Much)cited by 5