mlsanityData to 7 Oct 2026

MiniLM: Deep Self-Attention Distillation for Task-Agnostic Compression of Pre-Trained Transformers

arXiv 2002.10957 ↗First posted 25 Feb 2020cs.CLcited by 1,414

Not on the map. Why?

Cited by 124 in LLM agents and tool use, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

Pre-trained language models (e.g., BERT (Devlin et al., 2018) and its variants) have achieved remarkable success in varieties of NLP tasks. However, these models usually consist of hundreds of millions of parameters which brings challenges for fine-tuning and online serving in real-life applications due to latency and capacity constraints. In this work, we present a simple and effective approach to compress large Transformer (Vaswani et al., 2017) based pre-trained models, termed as deep self-attention distillation. The small model (student) is trained by deeply mimicking the self-attention module, which plays a vital role in Transformer networks, of the large model (teacher). Specifically, we propose distilling the self-attention module of the last Transformer layer of the teacher, which is effective and flexible for the student. Furthermore, we introduce the scaled dot-product between values in the self-attention module as the new deep self-attention knowledge, in addition to the attention distributions (i.e., the scaled dot-product of queries and keys) that have been used in existing works. Moreover, we show that introducing a teacher assistant (Mirzadeh et al., 2019) also helps the distillation of large pre-trained Transformer models. Experimental results demonstrate that our monolingual model outperforms state-of-the-art baselines in different parameter size of student models. In particular, it retains more than 99% accuracy on SQuAD 2.0 and several GLUE benchmark tasks using 50% of the Transformer parameters and computations of the teacher model. We also obtain competitive results in applying deep self-attention distillation to multilingual pre-trained models.

Key facts. Cited by 1,414 papers in the map (data through 7 Oct 2026). First posted on arXiv 25 Feb 2020.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 84; 2022: 126; 2023: 170; 2024: 229; 2025: 327; 2026: 478.

1,414 papers.

Sort:

By area:

  1. 2021BEIR: A Heterogenous Benchmark for Zero-shot Evaluation of Information Retrieval Modelscited by 1,225
  2. 2021DeBERTaV3: Improving DeBERTa using ELECTRA-Style Pre-Training with Gradient-Disentangled Embedding Sharingcited by 1,219
  3. 2023On-Policy Distillation of Language Models: Learning from Self-Generated Mistakescited by 840
  4. 2022GLM-130B: An Open Bilingual Pre-trained Modelcited by 837
  5. 2022Discovering Language Model Behaviors with Model-Written Evaluationscited by 805
  6. 2023Towards General Text Embeddings with Multi-stage Contrastive Learningcited by 705
  7. 2023LLM-Pruner: On the Structural Pruning of Large Language Modelscited by 643
  8. 2022MTEB: Massive Text Embedding Benchmarkcited by 623
  9. 2021CALVIN: A Benchmark for Language-Conditioned Policy Learning for Long-Horizon Robot Manipulation Taskscited by 564
  10. 2023MemoryBank: Enhancing Large Language Models with Long-Term Memorycited by 539
  11. 2022ZeroQuant: Efficient and Affordable Post-Training Quantization for Large-Scale Transformerscited by 517
  12. 2021ColBERTv2: Effective and Efficient Retrieval via Lightweight Late Interactioncited by 464
38 more
  1. 2022DeepSpeed-MoE: Advancing Mixture-of-Experts Inference and Training to Power Next-Generation AI Scalecited by 400
  2. 2024Navigating the Digital World as Humans Do: Universal Visual Grounding for GUI Agentscited by 373
  3. 2023Orca: Progressive Learning from Complex Explanation Traces of GPT-4cited by 369
  4. 2022Text and Code Embeddings by Contrastive Pre-Trainingcited by 358
  5. 2021Pre-Trained Models: Past, Present and Futurecited by 319
  6. 2022Confident Adaptive Language Modelingcited by 297
  7. 2021TrOCR: Transformer-based Optical Character Recognition with Pre-trained Modelscited by 278
  8. 2026Rethinking On-Policy Distillation of Large Language Models: Phenomenology, Mechanism, and Recipecited by 262
  9. 2021I-BERT: Integer-only BERT Quantizationcited by 250
  10. 2022Language model compression with weighted low-rank factorizationcited by 192
  11. 2024ColPali: Efficient Document Retrieval with Vision Language Modelscited by 190
  12. 2021FedKD: Communication Efficient Federated Learning via Knowledge Distillationcited by 185
  13. 2023MedCPT: Contrastive Pre-trained Transformers with Large-scale PubMed Search Logs for Zero-shot Biomedical Information Retrievalcited by 182
  14. 2022Structured Pruning Learns Compact and Accurate Modelscited by 167
  15. 2025Multi-agent Architecture Search via Agentic Supernetcited by 164
  16. 2023Speculative Decoding with Big Little Decodercited by 163
  17. 2022A Fast Post-Training Pruning Framework for Transformerscited by 161
  18. 2023Prompting4Debugging: Red-Teaming Text-to-Image Diffusion Models by Finding Problematic Promptscited by 161
  19. 2024OmniACT: A Dataset and Benchmark for Enabling Multimodal Generalist Autonomous Agents for Desktop and Webcited by 156
  20. 2023OpenWebMath: An Open Dataset of High-Quality Mathematical Web Textcited by 151
  21. 2026Entropy-Aware On-Policy Distillation of Language Modelscited by 151
  22. 2022Adapting Pre-trained Language Models to African Languages via Multilingual Adaptive Fine-Tuningcited by 149
  23. 2021AMMUS : A Survey of Transformer-based Pretrained Models in Natural Language Processingcited by 131
  24. 2023MiniLLM: On-Policy Distillation of Large Language Modelscited by 129
  25. 2023LaMini-LM: A Diverse Herd of Distilled Models from Large-Scale Instructionscited by 124
  26. 2024Hello Again! LLM-powered Personalized Agent for Long-term Dialoguecited by 121
  27. 2021mMARCO: A Multilingual Version of the MS MARCO Passage Ranking Datasetcited by 116
  28. 2022The Optimal BERT Surgeon: Scalable and Accurate Second-Order Pruning for Large Language Modelscited by 115
  29. 2023Prompting and Evaluating Large Language Models for Proactive Dialogues: Clarification, Target-guided, and Non-collaborationcited by 115
  30. 2025G-Memory: Tracing Hierarchical Memory for Multi-Agent Systemscited by 112
  31. 2024Knowledge Fusion of Large Language Modelscited by 110
  32. 2024G-Designer: Architecting Multi-agent Communication Topologies via Graph Neural Networkscited by 110
  33. 2023Towards Efficient Generative Large Language Model Serving: A Survey from Algorithms to Systemscited by 101
  34. 2024Can LLM be a Personalized Judge?cited by 99
  35. 2025Unveiling Privacy Risks in LLM Agent Memorycited by 99
  36. 2021Finetuning Pretrained Transformers into RNNscited by 94
  37. 2022Less is More: Task-aware Layer-wise Distillation for Language Model Compressioncited by 91
  38. 2026TIP: Token Importance in On-Policy Distillationcited by 90

50 most cited of 1,414 All 1,414 (JSON)