mlsanityData to 7 Oct 2026

Weight-Inherited Distillation for Task-Agnostic BERT Compression

arXiv 2305.09098 ↗First posted 16 May 2023cs.CL cs.LGcited by 8

Not on the map. Why?

Abstract (arXiv)

Knowledge Distillation (KD) is a predominant approach for BERT compression. Previous KD-based methods focus on designing extra alignment losses for the student model to mimic the behavior of the teacher model. These methods transfer the knowledge in an indirect way. In this paper, we propose a novel Weight-Inherited Distillation (WID), which directly transfers knowledge from the teacher. WID does not require any additional alignment loss and trains a compact student by inheriting the weights, showing a new perspective of knowledge distillation. Specifically, we design the row compactors and column compactors as mappings and then compress the weights via structural re-parameterization. Experimental results on the GLUE and SQuAD benchmarks show that WID outperforms previous state-of-the-art KD-based baselines. Further analysis indicates that WID can also learn the attention patterns from the teacher model without any alignment loss on attention distributions. The code is available at https://github.com/wutaiqiang/WID-NAACL2024.

Key facts. Cited by 8 papers in the map (data through 7 Oct 2026). Builds on 39 of its 51 references that are in the map. First posted on arXiv 16 May 2023.

Builds on

39 of 51 references.

References · 39

  1. 2017Attention Is All You Needcited by 59,932
  2. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  3. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  4. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,266
  5. 2015Distilling the Knowledge in a Neural Networkcited by 10,076
  6. 2016Layer Normalizationcited by 5,579
  7. 2018GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understandingcited by 4,854
  8. 2019DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lightercited by 4,243
  9. 2016SQuAD: 100,000+ Questions for Machine Comprehension of Textcited by 4,115
  10. 2019XLNet: Generalized Autoregressive Pretraining for Language Understandingcited by 2,801
  11. 2019ALBERT: A Lite BERT for Self-supervised Learning of Language Representationscited by 2,772
  12. 2017A Broad-Coverage Challenge Corpus for Sentence Understanding through Inferencecited by 2,665
27 more
  1. 2020Knowledge Distillation: A Surveycited by 1,578
  2. 2016Google's Neural Machine Translation System: Bridging the Gap between Human and Machine Translationcited by 1,471
  3. 2020MiniLM: Deep Self-Attention Distillation for Task-Agnostic Compression of Pre-Trained Transformerscited by 1,414
  4. 2019TinyBERT: Distilling BERT for Natural Language Understandingcited by 1,153
  5. 2015Aligning Books and Movies: Towards Story-like Visual Explanations by Watching Movies and Reading Bookscited by 1,041
  6. 2017SemEval-2017 Task 1: Semantic Textual Similarity - Multilingual and Cross-lingual Focused Evaluationcited by 998
  7. 2018Neural Network Acceptability Judgmentscited by 952
  8. 2019Patient Knowledge Distillation for BERT Model Compressioncited by 502
  9. 2019Be Your Own Teacher: Improve the Performance of Convolutional Neural Networks via Self Distillationcited by 491
  10. 2020MobileBERT: a Compact Task-Agnostic BERT for Resource-Limited Devicescited by 476
  11. 2020DynaBERT: Dynamic BERT with Adaptive Width and Depthcited by 246
  12. 2020MiniLMv2: Multi-Head Self-Attention Relation Distillation for Compressing Pretrained Transformerscited by 240
  13. 2021Block Pruning For Faster Transformerscited by 176
  14. 2022Structured Pruning Learns Compact and Accurate Modelscited by 167
  15. 2020Training with Quantization Noise for Extreme Model Compressioncited by 145
  16. 2020BERT-of-Theseus: Compressing BERT by Progressive Module Replacingcited by 112
  17. 2020Compressing Large-Scale Transformer-Based Models: A Case Study on BERTcited by 95
  18. 2020ResRep: Lossless CNN Pruning via Decoupling Remembering and Forgettingcited by 85
  19. 2022Compression of Generative Pre-trained Language Models via Quantizationcited by 79
  20. 2021BERT Learns to Teach: Knowledge Distillation with Meta Learningcited by 61
  21. 2020LadaBERT: Lightweight Adaptation of BERT through Hybrid Model Compressioncited by 45
  22. 2021Subformer: Exploring Weight Sharing for Parameter Efficiency in Generative Transformerscited by 42
  23. 2020Efficient Transformer-based Large Scale Language Representations using Hardware-friendly Block Structured Pruningcited by 39
  24. 2020Weight Distillation: Transferring the Knowledge in Neural Network Parameterscited by 23
  25. 2022TencentPretrain: A Scalable and Flexible Toolkit for Pre-training Models of Different Modalitiescited by 19
  26. 2023RIFormer: Keep Your Vision Backbone Effective While Removing Token Mixercited by 13
  27. 2023Edge-free but Structure-aware: Prototype-Guided Knowledge Distillation from GNNs to MLPscited by 7

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 1; 2024: 5; 2025: 1; 2026: 1.

8 papers.

  1. 2024Small Language Models: Survey, Measurements, and Insightscited by 109
  2. 2024Rethinking Kullback-Leibler Divergence in Knowledge Distillation for Large Language Modelscited by 60
  3. 2024Mixture-of-Subspaces in Low-Rank Adaptationcited by 47
  4. 2024A Survey on Transformer Compressioncited by 44
  5. 2023rm SP3: Enhancing Structured Pruning via PCA Projectioncited by 5
  6. 2026Diversity-Aware Reverse Kullback-Leibler Divergence for Large Language Model Distillationcited by 5
  7. 2024LoCa: Logit Calibration for Knowledge Distillationcited by 2
  8. 2025HaLoRA: Hardware-aware Low-Rank Adaptation for Large Language Models Based on Hybrid Compute-in-Memory Architecturecited by 2