# LoRAShear: Efficient Large Language Model Structured Pruning and Knowledge Recovery

arXiv: https://arxiv.org/abs/2310.18356 · HTML: https://mlsanity.com/paper/2310.18356/

## Key facts

- Cited by 38 papers in the map (data through 14 Oct 2026).
- Builds on 22 of its 28 references that are in the map.
- First posted on arXiv 24 Oct 2023.

## Builds on (most cited first, 22 of 22)

- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [LLaMA: Open and Efficient Foundation Language Models](https://mlsanity.com/paper/2302.13971/) (2023)
- [Sparks of Artificial General Intelligence: Early experiments with GPT-4](https://mlsanity.com/paper/2303.12712/) (2023)
- [A Survey of Large Language Models](https://mlsanity.com/paper/2303.18223/) (2023)
- [Emergent Abilities of Large Language Models](https://mlsanity.com/paper/2206.07682/) (2022)
- [Deep Compression: Compressing Deep Neural Networks with Pruning, Trained Quantization and Huffman Coding](https://mlsanity.com/paper/1510.00149/) (2015)
- [PaLM-E: An Embodied Multimodal Language Model](https://mlsanity.com/paper/2303.03378/) (2023)
- [SparseGPT: Massive Language Models Can Be Accurately Pruned in One-Shot](https://mlsanity.com/paper/2301.00774/) (2023)
- [Aligning Books and Movies: Towards Story-like Visual Explanations by Watching Movies and Reading Books](https://mlsanity.com/paper/1506.06724/) (2015)
- [A Simple and Effective Pruning Approach for Large Language Models](https://mlsanity.com/paper/2306.11695/) (2023)
- [LLM-Pruner: On the Structural Pruning of Large Language Models](https://mlsanity.com/paper/2305.11627/) (2023)
- [Sheared LLaMA: Accelerating Language Model Pre-training via Structured Pruning](https://mlsanity.com/paper/2310.06694/) (2023)
- [Learning to Prune Deep Neural Networks via Layer-wise Optimal Brain Surgeon](https://mlsanity.com/paper/1705.07565/) (2017)
- [Only Train Once: A One-Shot Neural Network Training And Pruning Framework](https://mlsanity.com/paper/2107.07467/) (2021)
- [A standardized Project Gutenberg corpus for statistical analysis of natural language and quantitative linguistics](https://mlsanity.com/paper/1812.08092/) (2018)
- [CDFI: Compression-Driven Network Design for Frame Interpolation](https://mlsanity.com/paper/2103.10559/) (2021)
- [OTOV2: Automatic, Generic, User-Friendly](https://mlsanity.com/paper/2303.06862/) (2023)
- [AGI Safety Literature Review](https://mlsanity.com/paper/1805.01109/) (2018)
- [Parameter-Efficient Sparsity for Large Language Models Fine-Tuning](https://mlsanity.com/paper/2205.11005/) (2022)
- [Orthant Based Proximal Stochastic Gradient Method for ℓ1-Regularized Optimization](https://mlsanity.com/paper/2004.03639/) (2020)
- [Neural Network Compression Via Sparse Optimization](https://mlsanity.com/paper/2011.04868/) (2020)
- [Sparsity-guided Network Design for Frame Interpolation](https://mlsanity.com/paper/2209.04551/) (2022)

## Built on by (most cited first, 30 of 38)

- [Assessing the Brittleness of Safety Alignment via Pruning and Low-Rank Modifications](https://mlsanity.com/paper/2402.05162/) (2024)
- [A Survey on Efficient Inference for Large Language Models](https://mlsanity.com/paper/2404.14294/) (2024)
- [A Survey on LoRA of Large Language Models](https://mlsanity.com/paper/2407.11046/) (2024)
- [Model Compression and Efficient Inference for Large Language Models: A Survey](https://mlsanity.com/paper/2402.09748/) (2024)
- [The Efficiency Spectrum of Large Language Models: An Algorithmic Survey](https://mlsanity.com/paper/2312.00678/) (2023)
- [Faster and Lighter LLMs: A Survey on Current Challenges and Way Forward](https://mlsanity.com/paper/2402.01799/) (2024)
- [SlimLLM: Accurate Structured Pruning for Large Language Models](https://mlsanity.com/paper/2505.22689/) (2025)
- [Automatic Joint Structured Pruning and Quantization for Efficient Neural Network Training and Compression](https://mlsanity.com/paper/2502.16638/) (2025)
- [Flexora: Flexible Low Rank Adaptation for Large Language Models](https://mlsanity.com/paper/2408.10774/) (2024)
- [LoRAP: Transformer Sub-Layers Deserve Differentiated Structured Compression for Large Language Models](https://mlsanity.com/paper/2404.09695/) (2024)
- [EfficientLLM: Scalable Pruning-Aware Pretraining for Architecture-Agnostic Edge Language Models](https://mlsanity.com/paper/2502.06663/) (2025)
- [Dynamic Low-Rank Sparse Adaptation for Large Language Models](https://mlsanity.com/paper/2502.14816/) (2025)
- [RILQ: Rank-Insensitive LoRA-based Quantization Error Compensation for Boosting 2-bit Large Language Model Accuracy](https://mlsanity.com/paper/2412.01129/) (2024)
- [NutePrune: Efficient Progressive Pruning with Numerous Teachers for Large Language Models](https://mlsanity.com/paper/2402.09773/) (2024)
- [OTOv3: Automatic Architecture-Agnostic Neural Network Training and Compression from Structured Pruning to Erasing Operators](https://mlsanity.com/paper/2312.09411/) (2023)
- [Bypass Back-propagation: Optimization-based Structural Pruning for Large Language Models via Policy Gradient](https://mlsanity.com/paper/2406.10576/) (2024)
- [Deep Hierarchical Learning with Nested Subspace Networks for Large Language Models](https://mlsanity.com/paper/2509.17874/) (2025)
- [HESSO: Towards Automatic Efficient and User Friendly Any Neural Network Training and Pruning](https://mlsanity.com/paper/2409.09085/) (2024)
- [Deploying Foundation Model Powered Agent Services: A Survey](https://mlsanity.com/paper/2412.13437/) (2024)
- [PASER: Post-Training Data Selection for Efficient Pruned Large Language Model Recovery](https://mlsanity.com/paper/2502.12594/) (2025)
- [Parallel Decoding via Hidden Transfer for Lossless Large Language Model Acceleration](https://mlsanity.com/paper/2404.12022/) (2024)
- [RAP: Runtime Adaptive Pruning for LLM Inference](https://mlsanity.com/paper/2505.17138/) (2025)
- [WINA: Weight Informed Neuron Activation for Accelerating Large Language Model Inference](https://mlsanity.com/paper/2505.19427/) (2025)
- [Why Lift so Heavy? Slimming Large Language Models by Cutting Off the Layers](https://mlsanity.com/paper/2402.11700/) (2024)
- [Mixture of Hidden-Dimensions Transformer](https://mlsanity.com/paper/2412.05644/) (2024)
- [LLM Inference Acceleration via Efficient Operation Fusion](https://mlsanity.com/paper/2502.17728/) (2025)
- [Sparsity Outperforms Low-Rank Projections in Few-Shot Adaptation](https://mlsanity.com/paper/2504.12436/) (2025)
- [MaskPro: Linear-Space Probabilistic Learning for Strict (N:M)-Sparsity on LLMs](https://mlsanity.com/paper/2506.12876/) (2025)
- [EcoSpa: Efficient Transformer Training with Coupled Sparsity](https://mlsanity.com/paper/2511.11641/) (2025)
- [WiSparse: Boosting LLM Inference Efficiency with Weight-Aware Mixed Activation Sparsity](https://mlsanity.com/paper/2602.14452/) (2026)

## Abstract (arXiv)

Large Language Models (LLMs) have transformed the landscape of artificial intelligence, while their enormous size presents significant challenges in terms of computational costs. We introduce LoRAShear, a novel efficient approach to structurally prune LLMs and recover knowledge. Given general LLMs, LoRAShear at first creates the dependency graphs over LoRA modules to discover minimally removal structures and analyze the knowledge distribution. It then proceeds progressive structured pruning on LoRA adaptors and enables inherent knowledge transfer to better preserve the information in the redundant structures. To recover the lost knowledge during pruning, LoRAShear meticulously studies and proposes a dynamic fine-tuning schemes with dynamic data adaptors to effectively narrow down the performance gap to the full models. Numerical results demonstrate that by only using one GPU within a couple of GPU days, LoRAShear effectively reduced footprint of LLMs by 20% with only 1.0% performance degradation and significantly outperforms state-of-the-arts. The source code will be available at https://github.com/microsoft/lorashear.