mlsanityData to 13 Oct 2026

LoRAShear: Efficient Large Language Model Structured Pruning and Knowledge Recovery

arXiv 2310.18356 ↗First posted 24 Oct 2023cs.CL cs.AI cs.LGcited by 38

Not on the map. Why?

Abstract (arXiv)

Large Language Models (LLMs) have transformed the landscape of artificial intelligence, while their enormous size presents significant challenges in terms of computational costs. We introduce LoRAShear, a novel efficient approach to structurally prune LLMs and recover knowledge. Given general LLMs, LoRAShear at first creates the dependency graphs over LoRA modules to discover minimally removal structures and analyze the knowledge distribution. It then proceeds progressive structured pruning on LoRA adaptors and enables inherent knowledge transfer to better preserve the information in the redundant structures. To recover the lost knowledge during pruning, LoRAShear meticulously studies and proposes a dynamic fine-tuning schemes with dynamic data adaptors to effectively narrow down the performance gap to the full models. Numerical results demonstrate that by only using one GPU within a couple of GPU days, LoRAShear effectively reduced footprint of LLMs by 20% with only 1.0% performance degradation and significantly outperforms state-of-the-arts. The source code will be available at https://github.com/microsoft/lorashear.

Key facts. Cited by 38 papers in the map (data through 13 Oct 2026). Builds on 22 of its 28 references that are in the map. First posted on arXiv 24 Oct 2023.

Builds on

22 of 28 references.

References · 22

  1. 2020Language Models are Few-Shot Learnerscited by 32,320
  2. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 14,072
  3. 2023Sparks of Artificial General Intelligence: Early experiments with GPT-4cited by 2,586
  4. 2023A Survey of Large Language Modelscited by 2,559
  5. 2022Emergent Abilities of Large Language Modelscited by 2,499
  6. 2015Deep Compression: Compressing Deep Neural Networks with Pruning, Trained Quantization and Huffman Codingcited by 2,315
  7. 2023PaLM-E: An Embodied Multimodal Language Modelcited by 2,052
  8. 2023SparseGPT: Massive Language Models Can Be Accurately Pruned in One-Shotcited by 1,103
  9. 2015Aligning Books and Movies: Towards Story-like Visual Explanations by Watching Movies and Reading Bookscited by 1,042
  10. 2023A Simple and Effective Pruning Approach for Large Language Modelscited by 787
  11. 2023LLM-Pruner: On the Structural Pruning of Large Language Modelscited by 646
  12. 2023Sheared LLaMA: Accelerating Language Model Pre-training via Structured Pruningcited by 409
10 more
  1. 2017Learning to Prune Deep Neural Networks via Layer-wise Optimal Brain Surgeoncited by 219
  2. 2021Only Train Once: A One-Shot Neural Network Training And Pruning Frameworkcited by 70
  3. 2018A standardized Project Gutenberg corpus for statistical analysis of natural language and quantitative linguisticscited by 64
  4. 2021CDFI: Compression-Driven Network Design for Frame Interpolationcited by 55
  5. 2023OTOV2: Automatic, Generic, User-Friendlycited by 32
  6. 2018AGI Safety Literature Reviewcited by 28
  7. 2022Parameter-Efficient Sparsity for Large Language Models Fine-Tuningcited by 27
  8. 2020Orthant Based Proximal Stochastic Gradient Method for ℓ1-Regularized Optimizationcited by 19
  9. 2020Neural Network Compression Via Sparse Optimizationcited by 12
  10. 2022Sparsity-guided Network Design for Frame Interpolationcited by 6

Built on by

Cited by year (2026 to 13 Oct)

Citing papers per year: 2023: 2; 2024: 17; 2025: 14; 2026: 5.

38 papers.

Sort:

By area:

  1. 2024Assessing the Brittleness of Safety Alignment via Pruning and Low-Rank Modificationscited by 227
  2. 2024A Survey on Efficient Inference for Large Language Modelscited by 202
  3. 2024A Survey on LoRA of Large Language Modelscited by 67
  4. 2024Model Compression and Efficient Inference for Large Language Models: A Surveycited by 64
  5. 2023The Efficiency Spectrum of Large Language Models: An Algorithmic Surveycited by 26
  6. 2024Faster and Lighter LLMs: A Survey on Current Challenges and Way Forwardcited by 21
  7. 2025SlimLLM: Accurate Structured Pruning for Large Language Modelscited by 18
  8. 2025Automatic Joint Structured Pruning and Quantization for Efficient Neural Network Training and Compressioncited by 14
  9. 2024Flexora: Flexible Low Rank Adaptation for Large Language Modelscited by 13
  10. 2024LoRAP: Transformer Sub-Layers Deserve Differentiated Structured Compression for Large Language Modelscited by 12
  11. 2025EfficientLLM: Scalable Pruning-Aware Pretraining for Architecture-Agnostic Edge Language Modelscited by 10
  12. 2025Dynamic Low-Rank Sparse Adaptation for Large Language Modelscited by 10
26 more
  1. 2024RILQ: Rank-Insensitive LoRA-based Quantization Error Compensation for Boosting 2-bit Large Language Model Accuracycited by 9
  2. 2024NutePrune: Efficient Progressive Pruning with Numerous Teachers for Large Language Modelscited by 7
  3. 2023OTOv3: Automatic Architecture-Agnostic Neural Network Training and Compression from Structured Pruning to Erasing Operatorscited by 6
  4. 2024Bypass Back-propagation: Optimization-based Structural Pruning for Large Language Models via Policy Gradientcited by 6
  5. 2025Deep Hierarchical Learning with Nested Subspace Networks for Large Language Modelscited by 5
  6. 2024HESSO: Towards Automatic Efficient and User Friendly Any Neural Network Training and Pruningcited by 4
  7. 2024Deploying Foundation Model Powered Agent Services: A Surveycited by 4
  8. 2024Parallel Decoding via Hidden Transfer for Lossless Large Language Model Accelerationcited by 3
  9. 2025PASER: Post-Training Data Selection for Efficient Pruned Large Language Model Recoverycited by 3
  10. 2025RAP: Runtime Adaptive Pruning for LLM Inferencecited by 3
  11. 2025WINA: Weight Informed Neuron Activation for Accelerating Large Language Model Inferencecited by 3
  12. 2024Why Lift so Heavy? Slimming Large Language Models by Cutting Off the Layerscited by 2
  13. 2024Mixture of Hidden-Dimensions Transformercited by 2
  14. 2025LLM Inference Acceleration via Efficient Operation Fusioncited by 1
  15. 2025Sparsity Outperforms Low-Rank Projections in Few-Shot Adaptationcited by 1
  16. 2025MaskPro: Linear-Space Probabilistic Learning for Strict (N:M)-Sparsity on LLMscited by 1
  17. 2025EcoSpa: Efficient Transformer Training with Coupled Sparsitycited by 1
  18. 2024Mixed Sparsity Training: Achieving 4× FLOP Reduction for Transformer Pretrainingcited by 0
  19. 2024Adaptive Pruning for Large Language Models with Structural Importance Awarenesscited by 0
  20. 20251+1>2: A Synergistic Sparse and Low-Rank Compression Method for Large Language Modelscited by 0
  21. 2025The Quest for Winning Tickets in Low-Rank Adapterscited by 0
  22. 2026WiSparse: Boosting LLM Inference Efficiency with Weight-Aware Mixed Activation Sparsitycited by 0
  23. 2026Diet Your LLM: Dimension-wise Global Pruning of LLMs via Merging Task-specific Importance Scorecited by 0
  24. 2026RecycleLoRA: Rank-Revealing QR-Based Dual-LoRA Subspace Adaptation for Domain Generalized Semantic Segmentationcited by 0
  25. 2026CascadeFormer: Depth-Tapered Transformers Motivated by Gradient Fan-in Asymmetrycited by 0
  26. 2026The Functional Structure of Post-Compression Recovery in Low-Rank LLMscited by 0