mlsanityData to 14 Oct 2026

Low-Rank Prune-And-Factorize for Language Model Compression

arXiv 2306.14152 ↗First posted 25 Jun 2023cs.CLcited by 11

Not on the map. Why?

Abstract (arXiv)

The components underpinning PLMs -- large weight matrices -- were shown to bear considerable redundancy. Matrix factorization, a well-established technique from matrix theory, has been utilized to reduce the number of parameters in PLM. However, it fails to retain satisfactory performance under moderate to high compression rate. In this paper, we identify the textitfull-rankness of fine-tuned PLM as the fundamental bottleneck for the failure of matrix factorization and explore the use of network pruning to extract low-rank sparsity pattern desirable to matrix factorization. We find such low-rank sparsity pattern exclusively exists in models generated by first-order pruning, which motivates us to unite the two approaches and achieve more effective model compression. We further propose two techniques: sparsity-aware SVD and mixed-rank fine-tuning, which improve the initialization and training of the compression procedure, respectively. Experiments on GLUE and question-answering tasks show that the proposed method has superior compression-performance trade-off compared to existing approaches.

Key facts. Cited by 11 papers in the map (data through 14 Oct 2026). Builds on 37 of its 47 references that are in the map. First posted on arXiv 25 Jun 2023.

Builds on

37 of 47 references.

References · 37

  1. 2017Attention Is All You Needcited by 60,361
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  3. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,599
  4. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,372
  5. 2015Distilling the Knowledge in a Neural Networkcited by 10,324
  6. 2019BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehensioncited by 5,893
  7. 2018GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understandingcited by 4,899
  8. 2019DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lightercited by 4,307
  9. 2016SQuAD: 100,000+ Questions for Machine Comprehension of Textcited by 4,148
  10. 2017A Broad-Coverage Challenge Corpus for Sentence Understanding through Inferencecited by 2,682
  11. 2013Estimating or Propagating Gradients Through Stochastic Neurons for Conditional Computationcited by 2,079
  12. 2015Learning both Weights and Connections for Efficient Neural Networkscited by 2,031
25 more
  1. 2017Decoupled Weight Decay Regularizationcited by 1,831
  2. 2018Know What You Don't Know: Unanswerable Questions for SQuADcited by 1,554
  3. 2020MiniLM: Deep Self-Attention Distillation for Task-Agnostic Compression of Pre-Trained Transformerscited by 1,463
  4. 2019TinyBERT: Distilling BERT for Natural Language Understandingcited by 1,167
  5. 2019Deep Double Descent: Where Bigger Models and More Data Hurtcited by 672
  6. 2017Learning Sparse Neural Networks through L0 Regularizationcited by 583
  7. 2019Patient Knowledge Distillation for BERT Model Compressioncited by 508
  8. 2020Movement Pruning: Adaptive Sparsity by Fine-Tuningcited by 403
  9. 2020The Lottery Ticket Hypothesis for Pre-trained BERT Networkscited by 275
  10. 2020DynaBERT: Dynamic BERT with Adaptive Width and Depthcited by 251
  11. 2020MiniLMv2: Multi-Head Self-Attention Relation Distillation for Compressing Pretrained Transformerscited by 245
  12. 2019Structured Pruning of Large Language Modelscited by 203
  13. 2021R-Drop: Regularized Dropout for Neural Networkscited by 202
  14. 2022Language model compression with weighted low-rank factorizationcited by 195
  15. 2021Block Pruning For Faster Transformerscited by 177
  16. 2022Structured Pruning Learns Compact and Accurate Modelscited by 170
  17. 2019Well-Read Students Learn Better: On the Importance of Pre-training Compact Modelscited by 120
  18. 2020BERT-of-Theseus: Compressing BERT by Progressive Module Replacingcited by 113
  19. 2021BERT Learns to Teach: Knowledge Distillation with Meta Learningcited by 61
  20. 2016Pruning Convolutional Neural Networks for Resource Efficient Inferencecited by 48
  21. 2021Distilling Linguistic Context for Language Model Compressioncited by 33
  22. 2018GroupReduce: Block-Wise Low-Rank Approximation for Neural Language Model Shrinkingcited by 32
  23. 2018Balanced Sparsity for Efficient DNN Inference on GPUcited by 27
  24. 2021RAIL-KD: RAndom Intermediate Layer Mapping for Knowledge Distillationcited by 19
  25. 2021From Dense to Sparse: Contrastive Pruning for Better Pre-trained Language Model Compressioncited by 19

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 2; 2024: 3; 2025: 4; 2026: 2.

11 papers.

  1. 2024Model Compression and Efficient Inference for Large Language Models: A Surveycited by 64
  2. 2023LORD: Low Rank Decomposition Of Monolingual Code LLMs For One-Shot Compressioncited by 26
  3. 2024LoRAP: Transformer Sub-Layers Deserve Differentiated Structured Compression for Large Language Modelscited by 12
  4. 2024Adaptive Feature-based Low-Rank Compression of Large Language Models via Bayesian Optimizationcited by 10
  5. 2025Forget the Data and Fine-Tuning! Just Fold the Network to Compresscited by 7
  6. 2023rm SP3: Enhancing Structured Pruning via PCA Projectioncited by 5
  7. 2026Cut Less, Fold More: Model Compression through the Lens of Projection Geometrycited by 4
  8. 2025MLoRQ: Bridging Low-Rank and Quantization for Transformer Compressioncited by 1
  9. 2025GradOT: Training-free Gradient-preserving Offsite-tuning for Large Language Modelscited by 0
  10. 2025Accelerating Attention with Basis Decompositioncited by 0
  11. 2026SPQ: An Ensemble Technique for Large Language Model Compressioncited by 0