mlsanityData to 14 Oct 2026

Exploring Extreme Parameter Compression for Pre-trained Language Models

arXiv 2205.10036 ↗First posted 20 May 2022cs.CL cs.LGcited by 15

Not on the map. Why?

Abstract (arXiv)

Recent work explored the potential of large-scale Transformer-based pre-trained models, especially Pre-trained Language Models (PLMs) in natural language processing. This raises many concerns from various perspectives, e.g., financial costs and carbon emissions. Compressing PLMs like BERT with negligible performance loss for faster inference and cheaper deployment has attracted much attention. In this work, we aim to explore larger compression ratios for PLMs, among which tensor decomposition is a potential but under-investigated one. Two decomposition and reconstruction protocols are further proposed to improve the effectiveness and efficiency during compression. Our compressed BERT with 1/7 parameters in Transformer layers performs on-par with, sometimes slightly better than the original BERT in GLUE benchmark. A tiny version achieves 96.7% performance of BERT-base with 1/48 encoder parameters (i.e., less than 2M parameters excluding the embedding layer) and 2.7 × faster on inference. To show that the proposed method is orthogonal to existing compression methods like knowledge distillation, we also explore the benefit of the proposed method on a distilled BERT.

Key facts. Cited by 15 papers in the map (data through 14 Oct 2026). Builds on 50 of its 68 references that are in the map. First posted on arXiv 20 May 2022.

Builds on

50 of 68 references.

References · 50

  1. 2017Attention Is All You Needcited by 60,361
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  3. 2020Language Models are Few-Shot Learnerscited by 32,419
  4. 2015Distilling the Knowledge in a Neural Networkcited by 10,324
  5. 2018GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understandingcited by 4,899
  6. 2019DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lightercited by 4,307
  7. 2018Neural Ordinary Differential Equationscited by 3,779
  8. 2016Gaussian Error Linear Units (GELUs)cited by 3,502
  9. 2021Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsitycited by 3,073
  10. 2019ALBERT: A Lite BERT for Self-supervised Learning of Language Representationscited by 2,790
  11. 2020Big Bird: Transformers for Longer Sequencescited by 1,684
  12. 2020Reformer: The Efficient Transformercited by 1,668
38 more
  1. 2020Linformer: Self-Attention with Linear Complexitycited by 1,474
  2. 2020Transformer Feed-Forward Layers Are Key-Value Memoriescited by 1,336
  3. 2019TinyBERT: Distilling BERT for Natural Language Understandingcited by 1,167
  4. 2019What Does BERT Look At? An Analysis of BERT's Attentioncited by 1,003
  5. 2019Analyzing Multi-Head Self-Attention: Specialized Heads Do the Heavy Lifting, the Rest Can Be Prunedcited by 856
  6. 2019Are Sixteen Heads Really Better than One?cited by 816
  7. 2020A Primer in BERTology: What we know about how BERT workscited by 815
  8. 2020Efficient Transformers: A Surveycited by 790
  9. 2021Carbon Emissions and Large Neural Network Trainingcited by 567
  10. 2019Patient Knowledge Distillation for BERT Model Compressioncited by 508
  11. 2021Attention is Not All You Need: Pure Attention Loses Rank Doubly Exponentially with Depthcited by 419
  12. 2014Exploiting Linear Structure Within Convolutional Networks for Efficient Evaluationcited by 306
  13. 2019Revealing the Dark Secrets of BERTcited by 301
  14. 2019A Multiscale Visualization of Attention in the Transformer Modelcited by 292
  15. 2020Inductive Biases for Deep Learning of Higher-Level Cognitioncited by 275
  16. 2020DynaBERT: Dynamic BERT with Adaptive Width and Depthcited by 251
  17. 2015Tensorizing Neural Networkscited by 250
  18. 2014Speeding-up Convolutional Neural Networks Using Fine-tuned CP-Decompositioncited by 203
  19. 2020BinaryBERT: Pushing the Limit of BERT Quantizationcited by 178
  20. 2021Block Pruning For Faster Transformerscited by 177
  21. 2020TernaryBERT: Distillation-aware Ultra-low Bit BERTcited by 148
  22. 2021PanGu-α: Large-scale Autoregressive Pretrained Chinese Language Models with Auto-parallel Computationcited by 134
  23. 2020BERT-of-Theseus: Compressing BERT by Progressive Module Replacingcited by 113
  24. 2019A Tensorized Transformer for Language Modelingcited by 86
  25. 2020Multi-Head Attention: Collaborate Instead of Concatenatecited by 60
  26. 2017Tensor-Train Recurrent Neural Networks for Video Classificationcited by 59
  27. 2021NAS-BERT: Task-Agnostic and Adaptive-Size BERT Compression with Neural Architecture Searchcited by 56
  28. 2020Masked Language Modeling for Proteins via Linearly Scalable Long-Context Transformerscited by 49
  29. 2020LadaBERT: Lightweight Adaptation of BERT through Hybrid Model Compressioncited by 45
  30. 2017Learning Compact Recurrent Neural Networks with Block-Term Tensor Decompositioncited by 33
  31. 2018Loss-aware Weight Quantization of Deep Networkscited by 31
  32. 2021AutoTinyBERT: Automatic Hyper-parameter Optimization for Efficient Pre-trained Language Modelscited by 31
  33. 2020DeFormer: Decomposing Pre-trained Transformers for Faster Question Answeringcited by 29
  34. 2021Enabling Lightweight Fine-tuning for Pre-trained Language Model Compression based on Matrix Product Operatorscited by 29
  35. 2019Compressing deep neural networks by matrix product operatorscited by 28
  36. 2019Tensorized Embedding Layers for Efficient Model Compressioncited by 27
  37. 2019On the Effectiveness of Low-Rank Matrix Factorization for LSTM Model Compressioncited by 14
  38. 2019word2ket: Space-efficient Word Embeddings inspired by Quantum Entanglementcited by 14

Built on by

Cited by year

Citing papers per year: 2022: 2; 2023: 6; 2024: 4; 2025: 3.

15 papers.

Sort:

By area:

  1. 2022FacT: Factor-Tuning for Lightweight Adaptation on Vision Transformercited by 133
  2. 2024Model Compression and Efficient Inference for Large Language Models: A Surveycited by 64
  3. 2023Tensor Networks Meet Neural Networks: A Survey and Future Perspectivescited by 32
  4. 2024Lessons and Insights from a Unifying Study of Parameter-Efficient Fine-Tuning (PEFT) in Visual Recognitioncited by 16
  5. 2022Multi-CLS BERT: An Efficient Alternative to Traditional Ensemblingcited by 14
  6. 2023COMCAT: Towards Efficient Compression and Customization of Attention-Based Vision Modelscited by 14
  7. 2025ReplaceMe: Network Simplification via Depth Pruning and Transformer Block Linearizationcited by 13
  8. 2023Efficiency is Not Enough: A Critical Perspective of Environmentally Sustainable AIcited by 12
  9. 2023Accurate Retraining-free Pruning for Pretrained Encoder-based Language Modelscited by 11
  10. 2024Geometry is All You Need: A Unified Taxonomy of Matrix and Tensor Factorization for Compression of Generative Language Modelscited by 4
  11. 2023Revisiting Offline Compression: Going Beyond Factorization-based Methods for Transformer Language Modelscited by 1
  12. 2023Pit One Against Many: Leveraging Attention-head Embeddings for Parameter-efficient Multi-head Attentioncited by 1
3 more
  1. 2024An Empirical Study of the Influence of Adversarial Fine-Tuning on Compressed Neural Networkscited by 1
  2. 2025The Structure of Relation Decoding Linear Operators in Large Language Modelscited by 1
  3. 2025AdaRing: Towards Ultra-Light Vision-Language Adaptation via Cross-Layer Tensor Ring Decompositioncited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2021Prune Once for All: Sparse Pre-Trained Language Models
  2. 2021A Short Study on Compressing Decoder-Based Language Models
  3. 2022Compressing Pre-trained Transformers via Low-Bit NxM Sparsity for Natural Language Understanding
  4. 2021KroneckerBERT: Learning Kronecker Decomposition for Pre-trained Language Models via Knowledge Distillation
  5. 2023Quantization-Aware and Tensor-Compressed Training of Transformers for Natural Language Understanding
  6. 2021ROSITA: Refined BERT cOmpreSsion with InTegrAted techniques
  7. 2023Improving Knowledge Distillation for BERT Models: Loss Functions, Mapping Methods, and Weight Tuning
  8. 2022Extreme Compression for Pre-trained Transformers Made Simple and Efficient