mlsanityData to 7 Oct 2026

Compressing Large-Scale Transformer-Based Models: A Case Study on BERT

arXiv 2002.11985 ↗First posted 27 Feb 2020cs.LG stat.MLcited by 95

Not on the map. Why?

Abstract (arXiv)

Pre-trained Transformer-based models have achieved state-of-the-art performance for various Natural Language Processing (NLP) tasks. However, these models often have billions of parameters, and, thus, are too resource-hungry and computation-intensive to suit low-capability devices or applications with strict latency requirements. One potential remedy for this is model compression, which has attracted a lot of research attention. Here, we summarize the research in compressing Transformers, focusing on the especially popular BERT model. In particular, we survey the state of the art in compression for BERT, we clarify the current best practices for compressing large-scale Transformer models, and we provide insights into the workings of various methods. Our categorization and analysis also shed light on promising future research directions for achieving lightweight, accurate, and generic NLP models.

Key facts. Cited by 95 papers in the map (data through 7 Oct 2026). First posted on arXiv 27 Feb 2020.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 18; 2022: 23; 2023: 25; 2024: 19; 2025: 7; 2026: 3.

95 papers.

Sort:

By area:

  1. 2021Chasing Sparsity in Vision Transformers: An End-to-End Explorationcited by 181
  2. 2022Structured Pruning Learns Compact and Accurate Modelscited by 167
  3. 2023Stop Uploading Test Data in Plain Text: Practical Strategies for Mitigating Data Contamination by Evaluation Benchmarkscited by 139
  4. 2021AMMUS : A Survey of Transformer-based Pretrained Models in Natural Language Processingcited by 131
  5. 2022Pretrained Language Models for Text Generation: A Surveycited by 121
  6. 2021BERT Busters: Outlier Dimensions that Disrupt Transformerscited by 110
  7. 2021Graph-Guided Network for Irregularly Sampled Multivariate Time Seriescited by 110
  8. 2023TinyCLIP: CLIP Distillation via Affinity Mimicking and Weight Inheritancecited by 90
  9. 2021Few-Shot Self-Rationalization with Natural Language Promptscited by 83
  10. 2022Unified Visual Transformer Compressioncited by 79
  11. 2024SliM-LLM: Salience-Driven Mixed-Precision Quantization for Large Language Modelscited by 61
  12. 2023Sparse MoE as the New Dropout: Scaling Dense and Self-Slimmable Transformerscited by 59
38 more
  1. 2021Subformer: Exploring Weight Sharing for Parameter Efficiency in Generative Transformerscited by 42
  2. 2021Rethinking Network Pruning -- under the Pre-train and Fine-tune Paradigmcited by 40
  3. 2024Accurate LoRA-Finetuning Quantization of LLMs via Information Retentioncited by 40
  4. 2023A Survey of Techniques for Optimizing Transformer Inferencecited by 35
  5. 2022Efficient Fine-Tuning of BERT Models on the Edgecited by 29
  6. 2021Enabling Lightweight Fine-tuning for Pre-trained Language Model Compression based on Matrix Product Operatorscited by 28
  7. 2023CoTFormer: A Chain-of-Thought Driven Architecture with Budget-Adaptive Computation Cost at Inferencecited by 27
  8. 2023Gradient-Free Structured Pruning with Unlabeled Datacited by 26
  9. 2025FastCache: Fast Caching for Diffusion Transformer Through Learnable Linear Approximationcited by 23
  10. 2021Layer-wise Model Pruning based on Mutual Informationcited by 19
  11. 2024Pruning via Merging: Compressing LLMs via Manifold Alignment Based Layer Mergingcited by 18
  12. 2023Transformers in Healthcare: A Surveycited by 17
  13. 2022Mokey: Enabling Narrow Fixed-Point Inference for Out-of-the-Box Floating-Point Transformer Modelscited by 16
  14. 2023Investigating Hallucinations in Pruned Large Language Models for Abstractive Summarizationcited by 16
  15. 2022Intriguing Properties of Compression on Multilingual Modelscited by 15
  16. 2024A Comprehensive Survey of Compression Algorithms for Language Modelscited by 15
  17. 2022Differentially Private Model Compressioncited by 13
  18. 2023Knowledge Distillation in Vision Transformers: A Critical Reviewcited by 13
  19. 2021Adapting by Pruning: A Case Study on BERTcited by 11
  20. 2021Greedy-layer Pruning: Speeding up Transformer Models for Natural Language Processingcited by 11
  21. 2022Sparse Teachers Can Be Dense with Knowledgecited by 11
  22. 2024LinguAlchemy: Fusing Typological and Geographical Elements for Unseen Language Generalizationcited by 10
  23. 2023Pitfalls in Language Models for Code Intelligence: A Taxonomy and Surveycited by 9
  24. 2024Transformer Explainer: Learning LLM Transformers with Interactive Visual Explanation and Experimentationcited by 9
  25. 2026Influence-Preserving Proxies for Gradient-Based Data Selection in LLM Fine-tuningcited by 9
  26. 2021When in Doubt, Summon the Titans: Efficient Inference with Large Modelscited by 8
  27. 2023Weight-Inherited Distillation for Task-Agnostic BERT Compressioncited by 8
  28. 2023Sparse Binary Transformers for Multivariate Time Series Modelingcited by 8
  29. 2022A Win-win Deal: Towards Sparse and Robust Pre-trained Language Modelscited by 7
  30. 2021Model Compression for Domain Adaptation through Causal Effect Estimationcited by 6
  31. 2022ACORT: A Compact Object Relation Transformer for Parameter Efficient Image Captioningcited by 6
  32. 2022Survey: Exploiting Data Redundancy for Optimization of Deep Learningcited by 6
  33. 2022HashFormers: Towards Vocabulary-independent Pre-trained Transformerscited by 6
  34. 2021Cross-Modal Coherence for Text-to-Image Retrievalcited by 5
  35. 2021An Ensemble of Pre-trained Transformer Models For Imbalanced Multiclass Malware Classificationcited by 5
  36. 2022Which Student is Best? A Comprehensive Knowledge Distillation Exam for Task-Specific BERT Modelscited by 5
  37. 2022MiniALBERT: Model Distillation via Parameter-Efficient Recursive Transformerscited by 5
  38. 2022Too Brittle To Touch: Comparing the Stability of Quantization and Distillation Towards Developing Lightweight Low-Resource MT Modelscited by 5

50 most cited of 95 All 95 (JSON)