# Efficient Transformer-based Large Scale Language Representations using Hardware-friendly Block Structured Pruning

arXiv: https://arxiv.org/abs/2009.08065 · HTML: https://mlsanity.com/paper/2009.08065/

## Key facts

- Cited by 39 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 17 Sep 2020.

## Built on by (most cited first, 30 of 39)

- [Efficiently Scaling Transformer Inference](https://mlsanity.com/paper/2211.05102/) (2022)
- [A Survey on Deep Neural Network Pruning-Taxonomy, Comparison, Analysis, and Recommendations](https://mlsanity.com/paper/2308.06767/) (2023)
- [SPViT: Enabling Faster Vision Transformers via Soft Token Pruning](https://mlsanity.com/paper/2112.13890/) (2021)
- [A Fast Post-Training Pruning Framework for Transformers](https://mlsanity.com/paper/2204.09656/) (2022)
- [Learned Token Pruning for Transformers](https://mlsanity.com/paper/2107.00910/) (2021)
- [Sparse is Enough in Scaling Transformers](https://mlsanity.com/paper/2111.12763/) (2021)
- [Model Compression and Efficient Inference for Large Language Models: A Survey](https://mlsanity.com/paper/2402.09748/) (2024)
- [A Survey of Techniques for Optimizing Transformer Inference](https://mlsanity.com/paper/2307.07982/) (2023)
- [What Matters In The Structured Pruning of Generative Language Models?](https://mlsanity.com/paper/2302.03773/) (2023)
- [Accelerating Framework of Transformer by Hardware Design and Model Compression Co-Optimization](https://mlsanity.com/paper/2110.10030/) (2021)
- [Pruning Self-attentions into Convolutional Layers in Single Path](https://mlsanity.com/paper/2111.11802/) (2021)
- [Sparse Progressive Distillation: Resolving Overfitting under Pretrain-and-Finetune Paradigm](https://mlsanity.com/paper/2110.08190/) (2021)
- [A Provably Effective Method for Pruning Experts in Fine-tuned Sparse Mixture-of-Experts](https://mlsanity.com/paper/2405.16646/) (2024)
- [Pruning Foundation Models for High Accuracy without Retraining](https://mlsanity.com/paper/2410.15567/) (2024)
- [From Dense to Sparse: Contrastive Pruning for Better Pre-trained Language Model Compression](https://mlsanity.com/paper/2112.07198/) (2021)
- [Binary Complex Neural Network Acceleration on FPGA](https://mlsanity.com/paper/2108.04811/) (2021)
- [Differentially Private Model Compression](https://mlsanity.com/paper/2206.01838/) (2022)
- [An Automatic and Efficient BERT Pruning for Edge AI Systems](https://mlsanity.com/paper/2206.10461/) (2022)
- [Toward Adaptive Large Language Models Structured Pruning via Hybrid-grained Weight Importance Assessment](https://mlsanity.com/paper/2403.10799/) (2024)
- [Dancing along Battery: Enabling Transformer with Run-time Reconfigurability on Mobile Devices](https://mlsanity.com/paper/2102.06336/) (2021)
- [RoRA: Efficient Fine-Tuning of LLM with Reliability Optimization for Rank Adaptation](https://mlsanity.com/paper/2501.04315/) (2025)
- [Weight-Inherited Distillation for Task-Agnostic BERT Compression](https://mlsanity.com/paper/2305.09098/) (2023)
- [Large Multimodal Model Compression via Efficient Pruning and Distillation at AntGroup](https://mlsanity.com/paper/2312.05795/) (2023)
- [Bridging Fairness and Environmental Sustainability in Natural Language Processing](https://mlsanity.com/paper/2211.04256/) (2022)
- [Accelerating ViT Inference on FPGA through Static and Dynamic Pruning](https://mlsanity.com/paper/2403.14047/) (2024)
- [F3OCUS -- Federated Finetuning of Vision-Language Foundation Models with Optimal Client Layer Updating Strategy via Multi-objective Meta-Heuristics](https://mlsanity.com/paper/2411.11912/) (2024)
- [Exploring Attention Map Reuse for Efficient Transformer Neural Networks](https://mlsanity.com/paper/2301.12444/) (2023)
- [RecycleGPT: An Autoregressive Language Model with Recyclable Module](https://mlsanity.com/paper/2308.03421/) (2023)
- [SpeechNet: Weakly Supervised, End-to-End Speech Recognition at Industrial Scale](https://mlsanity.com/paper/2211.11740/) (2022)
- [Only Send What You Need: Learning to Communicate Efficiently in Federated Multilingual Machine Translation](https://mlsanity.com/paper/2401.07456/) (2024)

## Abstract (arXiv)

Pre-trained large-scale language models have increasingly demonstrated high accuracy on many natural language processing (NLP) tasks. However, the limited weight storage and computational speed on hardware platforms have impeded the popularity of pre-trained models, especially in the era of edge computing. In this work, we propose an efficient transformer-based large-scale language representation using hardware-friendly block structure pruning. We incorporate the reweighted group Lasso into block-structured pruning for optimization. Besides the significantly reduced weight storage and computation, the proposed approach achieves high compression rates. Experimental results on different models (BERT, RoBERTa, and DistilBERT) on the General Language Understanding Evaluation (GLUE) benchmark tasks show that we achieve up to 5.0x with zero or minor accuracy degradation on certain task(s). Our proposed method is also orthogonal to existing compact pre-trained language models such as DistilBERT using knowledge distillation, since a further 1.79x average compression rate can be achieved on top of DistilBERT with zero or minor accuracy degradation. It is suitable to deploy the final compressed model on resource-constrained edge devices.