mlsanityData to 7 Oct 2026

rm SP3: Enhancing Structured Pruning via PCA Projection

arXiv 2308.16475 ↗First posted 31 Aug 2023cs.CL cs.AIcited by 5

Not on the map. Why?

Abstract (arXiv)

Structured pruning is a widely used technique for reducing the size of pre-trained language models (PLMs), but current methods often overlook the potential of compressing the hidden dimension (d) in PLMs, a dimension critical to model size and efficiency. This paper introduces a novel structured pruning approach, Structured Pruning with PCA Projection (SP3), targeting the effective reduction of d by projecting features into a space defined by principal components before masking. Extensive experiments on benchmarks (GLUE and SQuAD) show that SP3 can reduce d by 70%, compress 94% of the BERTbase model, maintain over 96% accuracy, and outperform other methods that compress d by 6% in accuracy at the same compression ratio. SP3 has also proven effective with other models, including OPT and Llama. Our data and code are available at an anonymous repo.

Key facts. Cited by 5 papers in the map (data through 7 Oct 2026). Builds on 49 of its 73 references that are in the map. First posted on arXiv 31 Aug 2023.

Builds on

49 of 73 references.

References · 49

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  2. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,160
  3. 2022Training language models to follow instructions with human feedbackcited by 15,712
  4. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  5. 2015Distilling the Knowledge in a Neural Networkcited by 10,076
  6. 2016Layer Normalizationcited by 5,579
  7. 2018GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understandingcited by 4,854
  8. 2018Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challengecited by 4,710
  9. 2021RoFormer: Enhanced Transformer with Rotary Position Embeddingcited by 4,581
  10. 2019DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lightercited by 4,243
  11. 2016SQuAD: 100,000+ Questions for Machine Comprehension of Textcited by 4,115
  12. 2019HellaSwag: Can a Machine Really Finish Your Sentence?cited by 4,018
37 more
  1. 2019PIQA: Reasoning about Physical Commonsense in Natural Languagecited by 2,818
  2. 2019HuggingFace's Transformers: State-of-the-art Natural Language Processingcited by 2,201
  3. 2019BoolQ: Exploring the Surprising Difficulty of Natural Yes/No Questionscited by 2,140
  4. 2018Can a Suit of Armor Conduct Electricity? A New Dataset for Open Book Question Answeringcited by 2,114
  5. 2019TinyBERT: Distilling BERT for Natural Language Understandingcited by 1,153
  6. 2019Analyzing Multi-Head Self-Attention: Specialized Heads Do the Heavy Lifting, the Rest Can Be Prunedcited by 844
  7. 2019Are Sixteen Heads Really Better than One?cited by 803
  8. 2017Learning Sparse Neural Networks through L0 Regularizationcited by 571
  9. 2019The State of Sparsity in Deep Neural Networkscited by 471
  10. 2019Reducing Transformer Depth on Demand with Structured Dropoutcited by 434
  11. 2020Movement Pruning: Adaptive Sparsity by Fine-Tuningcited by 398
  12. 2020DynaBERT: Dynamic BERT with Adaptive Width and Depthcited by 246
  13. 2019Structured Pruning of Large Language Modelscited by 201
  14. 2021Block Pruning For Faster Transformerscited by 176
  15. 2022A Fast Post-Training Pruning Framework for Transformerscited by 161
  16. 2017Bilateral Multi-Perspective Matching for Natural Language Sentencescited by 120
  17. 2020On the Effect of Dropping Layers of Pre-trained Transformer Modelscited by 118
  18. 2020When BERT Plays the Lottery, All Tickets Are Winningcited by 114
  19. 2020BERT-of-Theseus: Compressing BERT by Progressive Module Replacingcited by 112
  20. 2019A Tensorized Transformer for Language Modelingcited by 85
  21. 2020ResRep: Lossless CNN Pruning via Decoupling Remembering and Forgettingcited by 85
  22. 2022PLATON: Pruning Large Transformer Models with Upper Confidence Bound of Weight Importancecited by 79
  23. 2023Pre-RMSNorm and Pre-CRMSNorm Transformers: Equivalent and Efficient Pre-LN Transformerscited by 30
  24. 2018Semantic Sentence Matching with Densely-connected Recurrent and Co-attentive Informationcited by 28
  25. 2020BERT-EMD: Many-to-Many Layer Mapping for BERT Compression with Earth Mover's Distancecited by 27
  26. 2020LRC-BERT: Latent-representation Contrastive Knowledge Distillation for Natural Language Understandingcited by 27
  27. 2023Gradient-Free Structured Pruning with Unlabeled Datacited by 26
  28. 2023AD-KD: Attribution-Driven Knowledge Distillation for Language Model Compressioncited by 26
  29. 2020Weight Distillation: Transferring the Knowledge in Neural Network Parameterscited by 23
  30. 2020Pruning Redundant Mappings in Transformer Models via Spectral-Normalized Identity Priorcited by 23
  31. 2021From Dense to Sparse: Contrastive Pruning for Better Pre-trained Language Model Compressioncited by 18
  32. 2018A Low Effort Approach to Structured CNN Design Using PCAcited by 14
  33. 2021ROSITA: Refined BERT cOmpreSsion with InTegrAted techniquescited by 14
  34. 2022Numerical Optimizations for Weighted Low-rank Estimation on Language Modelcited by 14
  35. 2023COMCAT: Towards Efficient Compression and Customization of Attention-Based Vision Modelscited by 14
  36. 2023Low-Rank Prune-And-Factorize for Language Model Compressioncited by 11
  37. 2023Weight-Inherited Distillation for Task-Agnostic BERT Compressioncited by 8

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 2; 2026: 3.

5 papers.

  1. 2025QUAD: Quantization and Parameter-Efficient Tuning of LLM with Activation Decompositioncited by 4
  2. 2025A Simple Linear Patch Revives Layer-Pruned Large Language Modelscited by 4
  3. 2026IntraSlice: Towards High-Performance Structural Pruning with Block-Intra PCA for LLMscited by 0
  4. 2026Mixtures of Subspaces for Bandwidth Efficient Context Parallel Trainingcited by 0
  5. 2026On the Interaction Between Model Compression and Test-Time Adaptationcited by 0