mlsanityData to 7 Oct 2026

MiniLMv2: Multi-Head Self-Attention Relation Distillation for Compressing Pretrained Transformers

arXiv 2012.15828 ↗First posted 31 Dec 2020cs.CLcited by 240

Not on the map. Why?

Abstract (arXiv)

We generalize deep self-attention distillation in MiniLM (Wang et al., 2020) by only using self-attention relation distillation for task-agnostic compression of pretrained Transformers. In particular, we define multi-head self-attention relations as scaled dot-product between the pairs of query, key, and value vectors within each self-attention module. Then we employ the above relational knowledge to train the student model. Besides its simplicity and unified principle, more favorably, there is no restriction in terms of the number of student's attention heads, while most previous work has to guarantee the same head number between teacher and student. Moreover, the fine-grained self-attention relations tend to fully exploit the interaction knowledge learned by Transformer. In addition, we thoroughly examine the layer selection strategy for teacher models, rather than just relying on the last layer as in MiniLM. We conduct extensive experiments on compressing both monolingual and multilingual pretrained models. Experimental results demonstrate that our models distilled from base-size and large-size teachers (BERT, RoBERTa and XLM-R) outperform the state-of-the-art.

Key facts. Cited by 240 papers in the map (data through 7 Oct 2026). First posted on arXiv 31 Dec 2020.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 11; 2022: 29; 2023: 49; 2024: 52; 2025: 56; 2026: 43.

240 papers.

Sort:

By area:

  1. 2021DeBERTaV3: Improving DeBERTa using ELECTRA-Style Pre-Training with Gradient-Disentangled Embedding Sharingcited by 1,219
  2. 2022Text Embeddings by Weakly-Supervised Contrastive Pre-trainingcited by 1,216
  3. 2024Multilingual E5 Text Embeddings: A Technical Reportcited by 357
  4. 2022Language model compression with weighted low-rank factorizationcited by 192
  5. 2022Adapting Pre-trained Language Models to African Languages via Multilingual Adaptive Fine-Tuningcited by 149
  6. 2024Compact Language Models via Pruning and Knowledge Distillationcited by 142
  7. 2023MiniLLM: On-Policy Distillation of Large Language Modelscited by 129
  8. 2025X-Teaming: Multi-Turn Jailbreaks and Defenses with Adaptive Multi-Agentscited by 112
  9. 2021Finetuning Pretrained Transformers into RNNscited by 94
  10. 2022Less is More: Task-aware Layer-wise Distillation for Language Model Compressioncited by 91
  11. 2022MiniViT: Compressing Vision Transformers with Weight Multiplexingcited by 88
  12. 2024NV-Retriever: Improving text embedding models with effective hard-negative miningcited by 83
38 more
  1. 2024LLMs know their vulnerabilities: Uncover Safety Gaps through Natural Distribution Shiftscited by 80
  2. 2021BERT Learns to Teach: Knowledge Distillation with Meta Learningcited by 61
  3. 2024Towards Cross-Tokenizer Distillation: the Universal Logit Distillation Loss for LLMscited by 61
  4. 2024Arctic-Embed: Scalable, Efficient, and Accurate Text Embedding Modelscited by 60
  5. 2024Learning diverse attacks on large language models for robust red-teaming and safety tuningcited by 54
  6. 2025Black-Box On-Policy Distillation of Large Language Modelscited by 53
  7. 2022Sparsely Activated Mixture-of-Experts are Robust Multi-Task Learnerscited by 51
  8. 2022FairLex: A Multilingual Benchmark for Evaluating Fairness in Legal Text Processingcited by 48
  9. 2021ERNIE 3.0 Titan: Exploring Larger-scale Knowledge Enhanced Pre-training for Language Understanding and Generationcited by 45
  10. 2022A Survey on Model Compression and Acceleration for Pretrained Language Modelscited by 45
  11. 2023SemEval-2023 Task 12: Sentiment Analysis for African Languages (AfriSenti-SemEval)cited by 41
  12. 2023Towards the Law of Capacity Gap in Distilling Language Modelscited by 41
  13. 2022MoEBERT: from BERT to Mixture-of-Experts via Importance-Guided Adaptationcited by 37
  14. 2023UDAPDR: Unsupervised Domain Adaptation via LLM Prompting and Distillation of Rerankerscited by 37
  15. 2024Better Synthetic Data by Retrieving and Transforming Existing Datasetscited by 36
  16. 2025A Survey on Efficient Vision-Language-Action Modelscited by 34
  17. 2025Relational Transformer: Toward Zero-Shot Foundation Models for Relational Datacited by 32
  18. 2022A Closer Look at Self-Supervised Lightweight Vision Transformerscited by 31
  19. 2022Deep versus Wide: An Analysis of Student Architectures for Task-Agnostic Knowledge Distillation of Self-Supervised Speech Modelscited by 28
  20. 2024MiniPLM: Knowledge Distillation for Pre-Training Language Modelscited by 27
  21. 2023AD-KD: Attribution-Driven Knowledge Distillation for Language Model Compressioncited by 26
  22. 2024The Scandinavian Embedding Benchmarks: Comprehensive Assessment of Multilingual and Monolingual Text Embeddingcited by 26
  23. 2023HomoDistil: Homotopic Task-Agnostic Distillation of Pre-trained Transformerscited by 25
  24. 2023FrenchMedMCQA: A French Multiple-Choice Question Answering Dataset for Medical domaincited by 25
  25. 2023Lifting the Curse of Capacity Gap in Distilling Language Modelscited by 25
  26. 2024Dual-Space Knowledge Distillation for Large Language Modelscited by 25
  27. 2023Asking Clarification Questions to Handle Ambiguity in Open-Domain QAcited by 24
  28. 2025AutoSchemaKG: Autonomous Knowledge Graph Construction through Dynamic Schema Induction from Web-Scale Corporacited by 23
  29. 2022ConvLab-3: A Flexible Dialogue System Toolkit Based on a Unified Data Formatcited by 22
  30. 2021Distilled Dual-Encoder Model for Vision-Language Understandingcited by 21
  31. 2025ABKD: Pursuing a Proper Allocation of the Probability Mass in Knowledge Distillation via α-β-Divergencecited by 21
  32. 2025WAInjectBench: Benchmarking Prompt Injection Detections for Web Agentscited by 21
  33. 2024GASLITEing the Retrieval: Exploring Vulnerabilities in Dense Embedding-based Searchcited by 20
  34. 2022Extreme Compression for Pre-trained Transformers Made Simple and Efficientcited by 19
  35. 2024Recent advances in text embedding: A Comprehensive Review of Top-Performing Methods on the MTEB Benchmarkcited by 19
  36. 2021RAIL-KD: RAndom Intermediate Layer Mapping for Knowledge Distillationcited by 18
  37. 2023Generic-to-Specific Distillation of Masked Autoencoderscited by 18
  38. 2021NewsBERT: Distilling Pre-trained Language Model for Intelligent News Applicationcited by 17

50 most cited of 240 All 240 (JSON)