mlsanityData to 14 Oct 2026

Outlier Weighed Layerwise Sparsity (OWL): A Missing Secret Sauce for Pruning LLMs to High Sparsity

arXiv 2310.05175 ↗First posted 8 Oct 2023cs.LGcited by 165

Not on the map. Why?

Abstract (arXiv)

Large Language Models (LLMs), renowned for their remarkable performance across diverse domains, present a challenge when it comes to practical deployment due to their colossal model size. In response to this challenge, efforts have been directed toward the application of traditional network pruning techniques to LLMs, uncovering a massive number of parameters that can be pruned in one-shot without hurting performance. Prevailing LLM pruning strategies have consistently adhered to the practice of uniformly pruning all layers at equivalent sparsity, resulting in robust performance. However, this observation stands in contrast to the prevailing trends observed in the field of vision models, where non-uniform layerwise sparsity typically yields stronger results. To understand the underlying reasons for this disparity, we conduct a comprehensive study and discover a strong correlation with the emergence of activation outliers in LLMs. Inspired by this finding, we introduce a novel LLM pruning methodology that incorporates a tailored set of non-uniform layerwise sparsity ratios, termed as Outlier Weighed Layerwise sparsity (OWL). The sparsity ratio of OWL is proportional to the outlier ratio observed within each layer, facilitating a more effective alignment between layerwise weight sparsity and outlier ratios. Our empirical evaluation, conducted across the LLaMA-V1 family and OPT, spanning various benchmarks, demonstrates the distinct advantages offered by OWL over previous methods. For instance, OWL exhibits a remarkable performance gain, surpassing the state-of-the-art Wanda and SparseGPT by 61.22 and 6.80 perplexity at a high sparsity level of 70%, respectively, while delivering 2.6x end-to-end inference speed-up in the DeepSparse inference engine. Codes are available at https://github.com/luuyin/OWL.

Key facts. Cited by 165 papers in the map (data through 14 Oct 2026). Builds on 53 of its 69 references that are in the map. First posted on arXiv 8 Oct 2023.

Builds on

53 of 69 references.

References · 53

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  2. 2020Language Models are Few-Shot Learnerscited by 32,419
  3. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,599
  4. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,380
  5. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 14,132
  6. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 13,028
  7. 2018GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understandingcited by 4,899
  8. 2018Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challengecited by 4,886
  9. 2020Training data-efficient image transformers & distillation through attentioncited by 4,586
  10. 2019HellaSwag: Can a Machine Really Finish Your Sentence?cited by 4,100
  11. 2022A ConvNet for the 2020scited by 3,579
  12. 2022OPT: Open Pre-trained Transformer Language Modelscited by 3,512
38 more
  1. 2016Pointer Sentinel Mixture Modelscited by 3,096
  2. 2022Emergent Abilities of Large Language Modelscited by 2,512
  3. 2018The Lottery Ticket Hypothesis: Finding Sparse, Trainable Neural Networkscited by 2,262
  4. 2019BoolQ: Exploring the Surprising Difficulty of Natural Yes/No Questionscited by 2,183
  5. 2018Can a Suit of Armor Conduct Electricity? A New Dataset for Open Book Question Answeringcited by 2,164
  6. 2015Learning both Weights and Connections for Efficient Neural Networkscited by 2,031
  7. 2022SmoothQuant: Accurate and Efficient Post-Training Quantization for Large Language Modelscited by 1,386
  8. 2023AWQ: Activation-aware Weight Quantization for LLM Compression and Accelerationcited by 1,298
  9. 2023SparseGPT: Massive Language Models Can Be Accurately Pruned in One-Shotcited by 1,121
  10. 2022LLM.int8(): 8-bit Matrix Multiplication for Transformers at Scalecited by 906
  11. 2018SNIP: Single-shot Network Pruning based on Connection Sensitivitycited by 812
  12. 2023A Simple and Effective Pruning Approach for Large Language Modelscited by 800
  13. 2023LLM-Pruner: On the Structural Pruning of Large Language Modelscited by 650
  14. 2017To prune, or not to prune: exploring the efficacy of pruning for model compressioncited by 603
  15. 2021Carbon Emissions and Large Neural Network Trainingcited by 567
  16. 2023Are Emergent Abilities of Large Language Models a Mirage?cited by 523
  17. 2019Rigging the Lottery: Making All Tickets Winnerscited by 495
  18. 2019The State of Sparsity in Deep Neural Networkscited by 476
  19. 2020Picking Winning Tickets Before Training by Preserving Gradient Flowcited by 471
  20. 2020Movement Pruning: Adaptive Sparsity by Fine-Tuningcited by 403
  21. 2017Scalable Training of Artificial Neural Networks with Adaptive Sparse Connectivity inspired by Network Sciencecited by 328
  22. 2022Estimating the Carbon Footprint of BLOOM, a 176B Parameter Language Modelcited by 265
  23. 2021All Bark and No Bite: Rogue Dimensions in Transformer Language Models Obscure Representational Qualitycited by 156
  24. 2019Towards Optimal Structured CNN Pruning via Generative Adversarial Learningcited by 146
  25. 2020Sparse GPU Kernels for Deep Learningcited by 141
  26. 2020Layer-adaptive sparsity for the Magnitude-based Pruningcited by 126
  27. 2022The Optimal BERT Surgeon: Scalable and Accurate Second-Order Pruning for Large Language Modelscited by 115
  28. 2021BERT Busters: Outlier Dimensions that Disrupt Transformerscited by 112
  29. 2021Do We Actually Need Dense Over-Parameterization? In-Time Over-Parameterization in Sparse Trainingcited by 110
  30. 2022The Unreasonable Effectiveness of Random Pruning: Return of the Most Naive Baseline for Sparse Trainingcited by 85
  31. 2021Sparse Training via Boosting Pruning Plasticity with Neuroregenerationcited by 84
  32. 2023Compressing LLMs: The Truth is Rarely Pure and Never Simplecited by 62
  33. 2022Outliers Dimensions that Disrupt Transformers Are Driven by Frequencycited by 55
  34. 2022Mixed-Precision Neural Network Quantization via Learned Layer-wise Importancecited by 54
  35. 2023The Emergence of Essential Sparsity in Large Pre-trained Models: The Weights that Mattercited by 42
  36. 2021Leveraging redundancy in attention with Reuse Transformerscited by 33
  37. 2016A topological insight into restricted Boltzmann machinescited by 32
  38. 2020Rethinking the Value of Transformer Componentscited by 25

50 most cited of 53 All 53 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 2; 2024: 43; 2025: 68; 2026: 52.

165 papers.

Sort:

By area:

  1. 2024LLM Inference Unveiled: Survey and Roofline Model Insightscited by 152
  2. 2024Sequoia: Scalable, Robust, and Hardware-aware Speculative Decodingcited by 103
  3. 2024DELLA-Merging: Reducing Interference in Model Merging through Magnitude-Based Samplingcited by 81
  4. 2024Found in the Middle: How Language Models Use Long Contexts Better via Plug-and-Play Positional Encodingcited by 72
  5. 2024Antidote: Post-fine-tuning Safety Alignment for Large Language Models against Harmful Fine-tuningcited by 72
  6. 2024Model Compression and Efficient Inference for Large Language Models: A Surveycited by 64
  7. 2025The Curse of Depth in Large Language Modelscited by 62
  8. 2024Decoding Compressed Trust: Scrutinizing the Trustworthiness of Efficient LLMs Under Compressioncited by 57
  9. 2024MaskLLM: Learnable Semi-Structured Sparsity for Large Language Modelscited by 50
  10. 2024MoDeGPT: Modular Decomposition for Large Language Model Compressioncited by 48
  11. 2024Pruner-Zero: Evolving Symbolic Pruning Metric from scratch for Large Language Modelscited by 40
  12. 2024Mix-LN: Unleashing the Power of Deeper Layers by Combining Pre-LN and Post-LNcited by 37
38 more
  1. 2024BlockPruner: Fine-grained Pruning for Large Language Modelscited by 36
  2. 2024AlphaPruning: Using Heavy-Tailed Self Regularization Theory for Improved Layer-wise Pruning of Large Language Modelscited by 36
  3. 2025Systematic Outliers in Large Language Modelscited by 33
  4. 2024STBLLM: Breaking the 1-Bit Barrier with Structured Binary LLMscited by 31
  5. 2024STUN: Structured-Then-Unstructured Pruning for Scalable MoE Pruningcited by 30
  6. 2024MicroScopiQ: Accelerating Foundational Models through Outlier-Aware Microscaling Quantizationcited by 28
  7. 2023The Efficiency Spectrum of Large Language Models: An Algorithmic Surveycited by 26
  8. 2024From Low Rank Gradient Subspace Stabilization to Low-Rank Weights: Observations, Theories, and Applicationscited by 25
  9. 2026Flow caching for autoregressive video generationcited by 25
  10. 2024FFN-SkipLLM: A Hidden Gem for Autoregressive Decoding with Adaptive Feed Forward Skippingcited by 23
  11. 2024Pushing the Limits of Large Language Model Quantization via the Linearity Theoremcited by 21
  12. 2025R-Sparse: Rank-Aware Activation Sparsity for Efficient LLM Inferencecited by 21
  13. 2024OATS: Outlier-Aware Pruning Through Sparse and Low Rank Decompositioncited by 20
  14. 2024Is C4 Dataset Optimal for Pruning? An Investigation of Calibration Data for LLM Pruningcited by 20
  15. 2024EvoPress: Accurate Dynamic Model Compression via Evolutionary Searchcited by 20
  16. 2025Finding Fantastic Experts in MoEs: A Unified Study for Expert Dropping Strategies and Observationscited by 20
  17. 2025DarwinLM: Evolutionary Structured Pruning of Large Language Modelscited by 18
  18. 2025SlimLLM: Accurate Structured Pruning for Large Language Modelscited by 18
  19. 2024A Comprehensive Survey of Compression Algorithms for Language Modelscited by 15
  20. 2024Enabling High-Sparsity Foundational Llama Models with Efficient Pretraining and Deploymentcited by 15
  21. 2024Beware of Calibration Data for Pruning Large Language Modelscited by 15
  22. 2023PERP: Rethinking the Prune-Retrain Paradigm in the Era of LLMscited by 14
  23. 2025When Reasoning Meets Compression: Understanding the Effects of LLMs Compression on Large Reasoning Modelscited by 13
  24. 2025AlphaDecay: Module-wise Weight Decay for Heavy-Tailed Balancing in LLMscited by 13
  25. 20252SSP: A Two-Stage Framework for Structured Pruning of LLMscited by 12
  26. 2025LayerIF: Estimating Layer Quality for Large Language Models using Influence Functionscited by 12
  27. 2025LIFT the Veil for the Truth: Principal Weights Emerge after Rank Reduction for Reasoning-Focused Supervised Fine-Tuningcited by 12
  28. 2024Fast and Effective Weight Update for Pruned Large Language Modelscited by 11
  29. 2024CFSP: An Efficient Structured Pruning Framework for LLMs with Coarse-to-Fine Activation Informationcited by 11
  30. 2025ProxSparse: Regularized Learning of Semi-Structured Sparsity Masks for Pretrained LLMscited by 11
  31. 2025Neural Parameter Search for Slimmer Fine-Tuned Models and Better Transfercited by 11
  32. 2024Condense, Don't Just Prune: Enhancing Efficiency and Performance in MoE Layer Pruningcited by 10
  33. 2025Dynamic Low-Rank Sparse Adaptation for Large Language Modelscited by 10
  34. 2025Eigenspectrum Analysis of Neural Networks without Aspect Ratio Biascited by 10
  35. 2024Inference Optimization of Foundation Models on AI Acceleratorscited by 9
  36. 2025TAMP: Token-Adaptive Layerwise Pruning in Multimodal Large Language Modelscited by 9
  37. 2025Preserving LLM Capabilities through Calibration Data Curation: From Analysis to Optimizationcited by 9
  38. 2026EvoESAP: Non-Uniform Expert Pruning for Sparse MoEcited by 9

50 most cited of 165 All 165 (JSON)