mlsanityData to 14 Oct 2026

A Comparative Analysis of Task-Agnostic Distillation Methods for Compressing Transformer Language Models

arXiv 2310.08797 ↗First posted 13 Oct 2023cs.CL cs.AIcited by 6

Not on the map. Why?

Abstract (arXiv)

Large language models have become a vital component in modern NLP, achieving state of the art performance in a variety of tasks. However, they are often inefficient for real-world deployment due to their expensive inference costs. Knowledge distillation is a promising technique to improve their efficiency while retaining most of their effectiveness. In this paper, we reproduce, compare and analyze several representative methods for task-agnostic (general-purpose) distillation of Transformer language models. Our target of study includes Output Distribution (OD) transfer, Hidden State (HS) transfer with various layer mapping strategies, and Multi-Head Attention (MHA) transfer based on MiniLMv2. Through our extensive experiments, we study the effectiveness of each method for various student architectures in both monolingual (English) and multilingual settings. Overall, we show that MHA transfer based on MiniLMv2 is generally the best option for distillation and explain the potential reasons behind its success. Moreover, we show that HS transfer remains as a competitive baseline, especially under a sophisticated layer mapping strategy, while OD transfer consistently lags behind other approaches. Findings from this study helped us deploy efficient yet effective student models for latency-critical applications.

Key facts. Cited by 6 papers in the map (data through 14 Oct 2026). Builds on 49 of its 74 references that are in the map. First posted on arXiv 13 Oct 2023.

Builds on

49 of 74 references.

References · 49

  1. 2017Attention Is All You Needcited by 60,361
  2. 2015Deep Residual Learning for Image Recognitioncited by 52,096
  3. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  4. 2020Language Models are Few-Shot Learnerscited by 32,419
  5. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 14,132
  6. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,372
  7. 2015Distilling the Knowledge in a Neural Networkcited by 10,324
  8. 2020Scaling Laws for Neural Language Modelscited by 6,586
  9. 2019BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehensioncited by 5,893
  10. 2016Layer Normalizationcited by 5,669
  11. 2018GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understandingcited by 4,899
  12. 2019DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lightercited by 4,307
37 more
  1. 2019Unsupervised Cross-lingual Representation Learning at Scalecited by 3,935
  2. 2016Gaussian Error Linear Units (GELUs)cited by 3,502
  3. 2020MiniLM: Deep Self-Attention Distillation for Task-Agnostic Compression of Pre-Trained Transformerscited by 1,463
  4. 2019TinyBERT: Distilling BERT for Natural Language Understandingcited by 1,167
  5. 2019BERT Rediscovers the Classical NLP Pipelinecited by 1,059
  6. 2015Aligning Books and Movies: Towards Story-like Visual Explanations by Watching Movies and Reading Bookscited by 1,043
  7. 2018XNLI: Evaluating Cross-lingual Sentence Representationscited by 915
  8. 2019Improved Knowledge Distillation via Teacher Assistantcited by 573
  9. 2019Patient Knowledge Distillation for BERT Model Compressioncited by 508
  10. 2020MobileBERT: a Compact Task-Agnostic BERT for Resource-Limited Devicescited by 482
  11. 2019Reducing Transformer Depth on Demand with Structured Dropoutcited by 439
  12. 2019Q-BERT: Hessian Based Ultra Low Precision Quantization of BERTcited by 359
  13. 2019Q8BERT: Quantized 8Bit BERTcited by 300
  14. 2021I-BERT: Integer-only BERT Quantizationcited by 255
  15. 2020MiniLMv2: Multi-Head Self-Attention Relation Distillation for Compressing Pretrained Transformerscited by 245
  16. 2020FastBERT: a Self-distilling BERT with Adaptive Inference Timecited by 226
  17. 2020BinaryBERT: Pushing the Limit of BERT Quantizationcited by 178
  18. 2021Block Pruning For Faster Transformerscited by 177
  19. 2022Structured Pruning Learns Compact and Accurate Modelscited by 170
  20. 2020BERT-of-Theseus: Compressing BERT by Progressive Module Replacingcited by 113
  21. 2020Densely Guided Knowledge Distillation using Multiple Teacher Assistantscited by 83
  22. 2020ALP-KD: Attention-Based Layer Projection for Knowledge Distillationcited by 66
  23. 2020MixKD: Towards Efficient Distillation of Large-scale Language Modelscited by 65
  24. 2021Annealing Knowledge Distillationcited by 61
  25. 2021BERT Learns to Teach: Knowledge Distillation with Meta Learningcited by 61
  26. 2020LadaBERT: Lightweight Adaptation of BERT through Hybrid Model Compressioncited by 45
  27. 2022A Survey on Model Compression and Acceleration for Pretrained Language Modelscited by 45
  28. 2021Distilling Linguistic Context for Language Model Compressioncited by 33
  29. 2020BERT-EMD: Many-to-Many Layer Mapping for BERT Compression with Earth Mover's Distancecited by 27
  30. 2020Why Skip If You Can Combine: A Simple Knowledge Distillation Technique for Intermediate Layerscited by 24
  31. 2021RAIL-KD: RAndom Intermediate Layer Mapping for Knowledge Distillationcited by 19
  32. 2021XtremeDistilTransformers: Task Transfer for Task-agnostic Distillationcited by 18
  33. 2022Multilingual Representation Distillation with Contrastive Learningcited by 11
  34. 2023How to Distill your BERT: An Empirical Study on the Impact of Weight Initialisation and Distillation Objectivescited by 9
  35. 2020Improving Task-Agnostic BERT Distillation with Layer Mapping Searchcited by 8
  36. 2023Neural Architecture Search for Effective Teacher-Student Knowledge Transfer in Language Modelscited by 6
  37. 2023Revisiting Intermediate Layer Distillation for Compressing Language Models: An Overfitting Perspectivecited by 5

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 4; 2025: 1; 2026: 1.

6 papers.

  1. 2025Sparse Logit Sampling: Accelerating Knowledge Distillation in LLMscited by 16
  2. 2024INDUS: Effective and Efficient Language Models for Scientific Applicationscited by 6
  3. 2024Intermediate Distillation: Data-Efficient Distillation from Black-Box LLMs for Information Retrievalcited by 3
  4. 2024Efficient Encoder-Decoder Transformer Decoding for Decomposable Taskscited by 2
  5. 2026Efficient Knowledge Distillation for LLMs: Offline Top-K Logits and a Fused Chunked KL Losscited by 2
  6. 2024GPTA: Generative Prompt Tuning Assistant for Synergistic Downstream Neural Network Enhancement with LLMscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2021ERNIE-Tiny : A Progressive Distillation Framework for Pretrained Transformer Compression
  2. 2023Task-agnostic Distillation of Encoder-Decoder Language Models
  3. 2025On-the-Fly Adaptive Distillation of Transformer to Dual-State Linear Attention
  4. 2023HomoDistil: Homotopic Task-Agnostic Distillation of Pre-trained Transformers
  5. 2021Greedy-layer Pruning: Speeding up Transformer Models for Natural Language Processing
  6. 2024Joint Fine-tuning and Conversion of Pretrained Speech and Language Models towards Linear Complexity
  7. 2026Effective Distillation to Hybrid xLSTM Architectures
  8. 2026Transformer-Based Language Models Across Domain Verticals: Architectures, Applications and Critical Assessment