mlsanityData to 7 Oct 2026

Revisiting Intermediate-Layer Matching in Knowledge Distillation: Layer-Selection Strategy Doesn't Matter (Much)

arXiv 2502.04499 ↗First posted 6 Feb 2025cs.LG cs.AI cs.CLcited by 5

Not on the map. Why?

Abstract (arXiv)

Knowledge distillation (KD) is a popular method of transferring knowledge from a large "teacher" model to a small "student" model. Previous work has explored various layer-selection strategies (e.g., forward matching and in-order random matching) for intermediate-layer matching in KD, where a student layer is forced to resemble a certain teacher layer. In this work, we revisit such layer-selection strategies and observe an intriguing phenomenon that layer-selection strategy does not matter (much) in intermediate-layer matching -- even seemingly nonsensical matching strategies such as reverse matching still result in surprisingly good student performance. We provide an interpretation for this phenomenon by examining the angles between teacher layers viewed from the student's perspective. Our work sheds light on KD practice, as layer-selection strategies may not be the main focus of KD system design, and vanilla forward matching works well in most setups.

Key facts. Cited by 5 papers in the map (data through 7 Oct 2026). Builds on 26 of its 39 references that are in the map. First posted on arXiv 6 Feb 2025.

Builds on

26 of 39 references.

References · 26

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  2. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  3. 2015Distilling the Knowledge in a Neural Networkcited by 10,076
  4. 2019BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehensioncited by 5,874
  5. 2019DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lightercited by 4,243
  6. 2016SQuAD: 100,000+ Questions for Machine Comprehension of Textcited by 4,115
  7. 2017A Broad-Coverage Challenge Corpus for Sentence Understanding through Inferencecited by 2,665
  8. 2018CommonsenseQA: A Question Answering Challenge Targeting Commonsense Knowledgecited by 1,793
  9. 2020MiniLM: Deep Self-Attention Distillation for Task-Agnostic Compression of Pre-Trained Transformerscited by 1,414
  10. 2019TinyBERT: Distilling BERT for Natural Language Understandingcited by 1,153
  11. 2019Relational Knowledge Distillationcited by 873
  12. 2015Scheduled Sampling for Sequence Prediction with Recurrent Neural Networkscited by 779
14 more
  1. 2019Patient Knowledge Distillation for BERT Model Compressioncited by 502
  2. 2024ShortGPT: Layers in Large Language Models are More Redundant Than You Expectcited by 369
  3. 2020MiniLMv2: Multi-Head Self-Attention Relation Distillation for Compressing Pretrained Transformerscited by 240
  4. 2024Lessons from the Trenches on Reproducible Evaluation of Language Modelscited by 172
  5. 2020DART: Open-Domain Structured Data Record to Text Generationcited by 144
  6. 2023f-Divergence Minimization for Sequence-Level Knowledge Distillationcited by 112
  7. 2020Pre-trained Summarization Distillationcited by 70
  8. 2020ALP-KD: Attention-Based Layer Projection for Knowledge Distillationcited by 65
  9. 2020ENGINE: Energy-Based Inference Networks for Non-Autoregressive Machine Translationcited by 34
  10. 2021RAIL-KD: RAndom Intermediate Layer Mapping for Knowledge Distillationcited by 18
  11. 2024Sinkhorn Distance Minimization for Knowledge Distillationcited by 15
  12. 2022Teacher Forcing Recovers Reward Functions for Text Generationcited by 11
  13. 2023Revisiting Intermediate Layer Distillation for Compressing Language Models: An Overfitting Perspectivecited by 5
  14. 2024LLMR: Knowledge Distillation with a Large Language Model-Induced Rewardcited by 3

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 1; 2026: 4.

5 papers.

  1. 2025UHKD: A Unified Framework for Heterogeneous Knowledge Distillation via Frequency-Domain Representationscited by 1
  2. 2026Distilling Genomic Models for Efficient mRNA Representation Learning via Embedding Matchingcited by 0
  3. 2026A Study on Hidden Layer Distillation for Large Language Model Pre-Trainingcited by 0
  4. 2026Geometric Foundation Model Distillation for Efficient Lunar 3D Reconstructioncited by 0
  5. 2026What Visual Generators Need from Teachers: Rethinking Representation Alignmentcited by 0