mlsanityData to 7 Oct 2026

LoRAP: Transformer Sub-Layers Deserve Differentiated Structured Compression for Large Language Models

arXiv 2404.09695 ↗First posted 15 Apr 2024cs.LG cs.AI cs.CLcited by 12

Not on the map. Why?

Abstract (arXiv)

Large language models (LLMs) show excellent performance in difficult tasks, but they often require massive memories and computational resources. How to reduce the parameter scale of LLMs has become research hotspots. In this study, we make an important observation that the multi-head self-attention (MHA) sub-layer of Transformer exhibits noticeable low-rank structure, while the feed-forward network (FFN) sub-layer does not. With this regard, we design a mixed compression model, which organically combines Low-Rank matrix approximation And structured Pruning (LoRAP). For the MHA sub-layer, we propose an input activation weighted singular value decomposition method to strengthen the low-rank characteristic. Furthermore, we discover that the weight matrices in MHA sub-layer have different low-rank degrees. Thus, a novel parameter allocation scheme according to the discrepancy of low-rank degrees is devised. For the FFN sub-layer, we propose a gradient-free structured channel pruning method. During the pruning, we get an interesting finding that the least important 1% of parameter actually play a vital role in model performance. Extensive evaluations on zero-shot perplexity and zero-shot task classification indicate that our proposal is superior to previous structured compression rivals under multiple compression ratios.

Key facts. Cited by 12 papers in the map (data through 7 Oct 2026). Builds on 32 of its 46 references that are in the map. First posted on arXiv 15 Apr 2024.

Builds on

32 of 46 references.

References · 32

  1. 2020Language Models are Few-Shot Learnerscited by 32,199
  2. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  3. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  4. 2018Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challengecited by 4,710
  5. 2022Large Language Models are Zero-Shot Reasonerscited by 4,323
  6. 2019HellaSwag: Can a Machine Really Finish Your Sentence?cited by 4,018
  7. 2023QLoRA: Efficient Finetuning of Quantized LLMscited by 3,355
  8. 2016Pointer Sentinel Mixture Modelscited by 2,992
  9. 2019PIQA: Reasoning about Physical Commonsense in Natural Languagecited by 2,818
  10. 2022Emergent Abilities of Large Language Modelscited by 2,483
  11. 2018Can a Suit of Armor Conduct Electricity? A New Dataset for Open Book Question Answeringcited by 2,114
  12. 2015Learning both Weights and Connections for Efficient Neural Networkscited by 1,998
20 more
  1. 2022GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformerscited by 1,838
  2. 2023SparseGPT: Massive Language Models Can Be Accurately Pruned in One-Shotcited by 1,090
  3. 2015Aligning Books and Movies: Towards Story-like Visual Explanations by Watching Movies and Reading Bookscited by 1,041
  4. 2022GLM-130B: An Open Bilingual Pre-trained Modelcited by 837
  5. 2023A Simple and Effective Pruning Approach for Large Language Modelscited by 776
  6. 2023LLM-Pruner: On the Structural Pruning of Large Language Modelscited by 643
  7. 2022ZeroQuant: Efficient and Affordable Post-Training Quantization for Large-Scale Transformerscited by 517
  8. 2020Movement Pruning: Adaptive Sparsity by Fine-Tuningcited by 398
  9. 2023LoftQ: LoRA-Fine-Tuning-Aware Quantization for Large Language Modelscited by 195
  10. 2022Language model compression with weighted low-rank factorizationcited by 192
  11. 2022Structured Pruning Learns Compact and Accurate Modelscited by 167
  12. 2022The Optimal BERT Surgeon: Scalable and Accurate Second-Order Pruning for Large Language Modelscited by 115
  13. 2023SCOTT: Self-Consistent Chain-of-Thought Distillationcited by 103
  14. 2023LoSparse: Structured Compression of Large Language Models based on Low-Rank and Sparse Approximationcited by 103
  15. 2023LoRAShear: Efficient Large Language Model Structured Pruning and Knowledge Recoverycited by 37
  16. 2023Beyond Size: How Gradients Shape Pruning Decisions in Large Language Modelscited by 30
  17. 2023LORD: Low Rank Decomposition Of Monolingual Code LLMs For One-Shot Compressioncited by 26
  18. 2022Numerical Optimizations for Weighted Low-rank Estimation on Language Modelcited by 14
  19. 2023Low-Rank Prune-And-Factorize for Language Model Compressioncited by 11
  20. 2022Gradient-based Intra-attention Pruning on Pre-trained Language Modelscited by 9

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 3; 2025: 5; 2026: 4.

12 papers.

  1. 2024Towards Efficient Mixture of Experts: A Holistic Study of Compression Techniquescited by 24
  2. 2024OATS: Outlier-Aware Pruning Through Sparse and Low Rank Decompositioncited by 19
  3. 2025DarwinLM: Evolutionary Structured Pruning of Large Language Modelscited by 16
  4. 2025Dynamic Low-Rank Sparse Adaptation for Large Language Modelscited by 10
  5. 2025CR-Net: Scaling Parameter-Efficient Training with Cross-Layer Low-Rank Structurecited by 7
  6. 2025Pangu Light: Weight Re-Initialization for Pruning and Accelerating LLMscited by 3
  7. 2024LORTSAR: Low-Rank Transformer for Skeleton-based Action Recognitioncited by 2
  8. 2026High-Fidelity Pruning for Large Language Modelscited by 1
  9. 2025MGAA: Multi-Granular Adaptive Allocation fof Low-Rank Compression of LLMscited by 0
  10. 2026UniRank: Unified Rank Allocation for Low-Rank LLM Compressioncited by 0
  11. 2026Structured Pruning of Large Language Models via Power Transformation and Sign-Preserving Score Aggregation with Adaptive Feature Retentioncited by 0
  12. 2026Sequential Functional Structured Tucker Compression for Large Language Model Attentionscited by 0