mlsanityData to 7 Oct 2026

How to Distill your BERT: An Empirical Study on the Impact of Weight Initialisation and Distillation Objectives

arXiv 2305.15032 ↗First posted 24 May 2023cs.CL cs.AI cs.LGcited by 9

Not on the map. Why?

Abstract (arXiv)

Recently, various intermediate layer distillation (ILD) objectives have been shown to improve compression of BERT models via Knowledge Distillation (KD). However, a comprehensive evaluation of the objectives in both task-specific and task-agnostic settings is lacking. To the best of our knowledge, this is the first work comprehensively evaluating distillation objectives in both settings. We show that attention transfer gives the best performance overall. We also study the impact of layer choice when initializing the student from the teacher layers, finding a significant impact on the performance in task-specific distillation. For vanilla KD and hidden states transfer, initialisation with lower layers of the teacher gives a considerable improvement over higher layers, especially on the task of QNLI (up to an absolute percentage change of 17.8 in accuracy). Attention transfer behaves consistently under different initialisation settings. We release our code as an efficient transformer-based model distillation framework for further studies.

Key facts. Cited by 9 papers in the map (data through 7 Oct 2026). Builds on 25 of its 28 references that are in the map. First posted on arXiv 24 May 2023.

Builds on

25 of 28 references.

References · 25

  1. 2017Attention Is All You Needcited by 59,932
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  3. 2020Language Models are Few-Shot Learnerscited by 32,199
  4. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,266
  5. 2015Distilling the Knowledge in a Neural Networkcited by 10,076
  6. 2018GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understandingcited by 4,854
  7. 2019DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lightercited by 4,243
  8. 2019XLNet: Generalized Autoregressive Pretraining for Language Understandingcited by 2,801
  9. 2019fairseq: A Fast, Extensible Toolkit for Sequence Modelingcited by 1,486
  10. 2020MiniLM: Deep Self-Attention Distillation for Task-Agnostic Compression of Pre-Trained Transformerscited by 1,414
  11. 2019TinyBERT: Distilling BERT for Natural Language Understandingcited by 1,153
  12. 2015Aligning Books and Movies: Towards Story-like Visual Explanations by Watching Movies and Reading Bookscited by 1,041
13 more
  1. 2019What Does BERT Look At? An Analysis of BERT's Attentioncited by 994
  2. 2019Patient Knowledge Distillation for BERT Model Compressioncited by 502
  3. 2020MobileBERT: a Compact Task-Agnostic BERT for Resource-Limited Devicescited by 476
  4. 2020MiniLMv2: Multi-Head Self-Attention Relation Distillation for Compressing Pretrained Transformerscited by 240
  5. 2022Knowledge Distillation from A Stronger Teachercited by 204
  6. 2021Comparing Kullback-Leibler Divergence and Mean Squared Error Loss in Knowledge Distillationcited by 131
  7. 2021How to Train BERT with an Academic Budgetcited by 80
  8. 2020ALP-KD: Attention-Based Layer Projection for Knowledge Distillationcited by 65
  9. 2020Wasserstein Contrastive Representation Distillationcited by 63
  10. 2020Contrastive Distillation on Intermediate Representations for Language Model Compressioncited by 57
  11. 2020Why Skip If You Can Combine: A Simple Knowledge Distillation Technique for Intermediate Layerscited by 24
  12. 2022Knowledge Distillation of Transformer-based Language Models Revisitedcited by 11
  13. 2022AutoDistill: an End-to-End Framework to Explore and Distill Hardware-Efficient Language Modelscited by 7

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 1; 2024: 3; 2025: 2; 2026: 3.

9 papers.

  1. 2024FinerCut: Finer-grained Interpretable Layer Pruning for Large Language Modelscited by 26
  2. 2025Sparse Logit Sampling: Accelerating Knowledge Distillation in LLMscited by 14
  3. 2023A Comparative Analysis of Task-Agnostic Distillation Methods for Compressing Transformer Language Modelscited by 6
  4. 2025Feature Alignment and Representation Transfer in Knowledge Distillation for Large Language Modelscited by 6
  5. 2026Sustained Gradient Alignment Mediates Subliminal Learning in a Multi-Step Setting: Evidence from MNIST Auxiliary Logit Distillation Experimentcited by 5
  6. 2024Why Lift so Heavy? Slimming Large Language Models by Cutting Off the Layerscited by 2
  7. 2024Knowledge Distillation vs. Pretraining from Scratch under a Fixed (Computation) Budgetcited by 1
  8. 2026Long-Context Encoder Models for Polish Language Understandingcited by 1
  9. 2026Teach Yourself Where to Look: On-Policy Attention Self-Distillation for Reasoningcited by 0