mlsanityData to 7 Oct 2026

LadaBERT: Lightweight Adaptation of BERT through Hybrid Model Compression

arXiv 2004.04124 ↗First posted 8 Apr 2020cs.CL cs.LGcited by 45

Not on the map. Why?

Abstract (arXiv)

BERT is a cutting-edge language representation model pre-trained by a large corpus, which achieves superior performances on various natural language understanding tasks. However, a major blocking issue of applying BERT to online services is that it is memory-intensive and leads to unsatisfactory latency of user requests, raising the necessity of model compression. Existing solutions leverage the knowledge distillation framework to learn a smaller model that imitates the behaviors of BERT. However, the training procedure of knowledge distillation is expensive itself as it requires sufficient training data to imitate the teacher model. In this paper, we address this issue by proposing a hybrid solution named LadaBERT (Lightweight adaptation of BERT through hybrid model compression), which combines the advantages of different model compression methods, including weight pruning, matrix factorization and knowledge distillation. LadaBERT achieves state-of-the-art accuracy on various public datasets while the training overheads can be reduced by an order of magnitude.

Key facts. Cited by 45 papers in the map (data through 7 Oct 2026). First posted on arXiv 8 Apr 2020.

Builds on

No reference list yet.

Built on by

Cited by year

Citing papers per year: 2021: 11; 2022: 12; 2023: 16; 2024: 2; 2025: 4.

45 papers.

Sort:

By area:

  1. 2022ZeroQuant: Efficient and Affordable Post-Training Quantization for Large-Scale Transformerscited by 517
  2. 2021I-BERT: Integer-only BERT Quantizationcited by 250
  3. 2023A Survey on Deep Neural Network Pruning-Taxonomy, Comparison, Analysis, and Recommendationscited by 200
  4. 2021Block Pruning For Faster Transformerscited by 176
  5. 2021Applications and Techniques for Fast Machine Learning in Sciencecited by 32
  6. 2023Matrix Compression via Randomized Low Rank and Low Precision Factorizationcited by 25
  7. 2023Understanding INT4 Quantization for Transformer Models: Latency Speedup, Composability, and Failure Casescited by 23
  8. 2021KroneckerBERT: Learning Kronecker Decomposition for Pre-trained Language Models via Knowledge Distillationcited by 19
  9. 2021NxMTransformer: Semi-Structured Sparsification for Natural Language Understanding via ADMMcited by 19
  10. 2022Extreme Compression for Pre-trained Transformers Made Simple and Efficientcited by 19
  11. 2022Efficient Quantized Sparse Matrix Operations on Tensor Corescited by 16
  12. 2022Exploring Extreme Parameter Compression for Pre-trained Language Modelscited by 15
33 more
  1. 2023TensorGPT: Efficient Compression of Large Language Models based on Tensor-Train Decompositioncited by 15
  2. 2021KDLSQ-BERT: A Quantized Bert Combining Knowledge Distillation with Learned Step Size Quantizationcited by 14
  3. 2021ROSITA: Refined BERT cOmpreSsion with InTegrAted techniquescited by 14
  4. 2022Differentially Private Model Compressioncited by 13
  5. 2022Learning to Win Lottery Tickets in BERT Transfer via Task-agnostic Mask Trainingcited by 10
  6. 2023Weight-Inherited Distillation for Task-Agnostic BERT Compressioncited by 8
  7. 2022Task-specific Compression for Multi-task Language Models using Attribution-based Pruningcited by 7
  8. 2022A Win-win Deal: Towards Sparse and Robust Pre-trained Language Modelscited by 7
  9. 2023Training Large Language Models Efficiently with Sparsity and Dataflowcited by 7
  10. 2022Bottleneck Low-rank Transformers for Low-resource Spoken Language Understandingcited by 6
  11. 2023A Comparative Analysis of Task-Agnostic Distillation Methods for Compressing Transformer Language Modelscited by 6
  12. 2023Revisiting Intermediate Layer Distillation for Compressing Language Models: An Overfitting Perspectivecited by 5
  13. 2024Cross-layer Attention Sharing for Pre-trained Large Language Modelscited by 5
  14. 2021Direction is what you need: Improving Word Embedding Compression in Large Language Modelscited by 4
  15. 2022Fine- and Coarse-Granularity Hybrid Self-Attention for Efficient BERTcited by 4
  16. 2022Vision Transformer Compression with Structured Pruning and Low Rank Approximationcited by 4
  17. 2023DSFormer: Effective Compression of Text-Transformers by Dense-Sparse Weight Factorizationcited by 3
  18. 2025Stratified Knowledge-Density Super-Network for Scalable Vision Transformerscited by 3
  19. 2022Compressing Pre-trained Transformers via Low-Bit NxM Sparsity for Natural Language Understandingcited by 2
  20. 2023Teacher Intervention: Improving Convergence of Quantization Aware Training for Ultra-Low Precision Transformerscited by 2
  21. 2023Quantization-Aware and Tensor-Compressed Training of Transformers for Natural Language Understandingcited by 2
  22. 2021General Cross-Architecture Distillation of Pretrained Language Models into Matrix Embeddingscited by 1
  23. 2023Revisiting Offline Compression: Going Beyond Factorization-based Methods for Transformer Language Modelscited by 1
  24. 2023Enhancing E-Commerce Recommendation using Pre-Trained Language Model and Fine-Tuningcited by 1
  25. 2023On Achieving Privacy-Preserving State-of-the-Art Edge Intelligencecited by 1
  26. 2023Pit One Against Many: Leveraging Attention-head Embeddings for Parameter-efficient Multi-head Attentioncited by 1
  27. 2025CURing Large Models: Compression via CUR Decompositioncited by 1
  28. 2021Classification-based Quality Estimation: Small and Efficient Models for Real-world Applicationscited by 0
  29. 2021BERMo: What can BERT learn from ELMo?cited by 0
  30. 2023Neural Language Model Pruning for Automatic Speech Recognitioncited by 0
  31. 2024Fully-fused Multi-Layer Perceptrons on Intel Data Center GPUscited by 0
  32. 2025TensorSLM: Energy-efficient Embedding Compression of Sub-billion Parameter Language Models on Low-end Devicescited by 0
  33. 2025Semantic Compression for Word and Sentence Embeddings using Discrete Wavelet Transformcited by 0