mlsanityData to 7 Oct 2026

OATS: Outlier-Aware Pruning Through Sparse and Low Rank Decomposition

arXiv 2409.13652 ↗First posted 20 Sep 2024cs.LG cs.AIcited by 19

Not on the map. Why?

Abstract (arXiv)

The recent paradigm shift to large-scale foundation models has brought about a new era for deep learning that, while has found great success in practice, has also been plagued by prohibitively expensive costs in terms of high memory consumption and compute. To mitigate these issues, there has been a concerted effort in post-hoc neural network pruning techniques that do not require costly retraining. Despite the considerable progress being made, existing methods often exhibit a steady drop in model performance as the compression increases. In this paper, we present a novel approach to compressing large transformers, coined OATS, that utilizes the second moment information in the input embeddings to decompose the model weights into a sum of sparse and low-rank matrices. Without any retraining, OATS achieves state-of-the-art performance when compressing models by up to 60% on large language models such as Llama-3 and Phi-3 and vision transformers such as ViT and DINOv2 while delivering up to 1.37× the CPU acceleration versus a model that was comparably pruned.

Key facts. Cited by 19 papers in the map (data through 7 Oct 2026). Builds on 50 of its 70 references that are in the map. First posted on arXiv 20 Sep 2024.

Builds on

50 of 70 references.

References · 50

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  2. 2020Language Models are Few-Shot Learnerscited by 32,199
  3. 2014Microsoft COCO: Common Objects in Contextcited by 17,498
  4. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  5. 2014ImageNet Large Scale Visual Recognition Challengecited by 8,441
  6. 2020Measuring Massive Multitask Language Understandingcited by 7,768
  7. 2018GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understandingcited by 4,854
  8. 2018Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challengecited by 4,710
  9. 2019HellaSwag: Can a Machine Really Finish Your Sentence?cited by 4,018
  10. 2019PIQA: Reasoning about Physical Commonsense in Natural Languagecited by 2,818
  11. 2019BoolQ: Exploring the Surprising Difficulty of Natural Yes/No Questionscited by 2,140
  12. 2018Can a Suit of Armor Conduct Electricity? A New Dataset for Open Book Question Answeringcited by 2,114
38 more
  1. 2023SparseGPT: Massive Language Models Can Be Accurately Pruned in One-Shotcited by 1,090
  2. 2023Vision Transformers Need Registerscited by 844
  3. 2018SNIP: Single-shot Network Pruning based on Connection Sensitivitycited by 802
  4. 2023A Simple and Effective Pruning Approach for Large Language Modelscited by 776
  5. 2020Quantifying Attention Flow in Transformerscited by 759
  6. 2023LLM-Pruner: On the Structural Pruning of Large Language Modelscited by 643
  7. 2021ImageNet-21K Pretraining for the Massescited by 607
  8. 2017To prune, or not to prune: exploring the efficacy of pruning for model compressioncited by 596
  9. 2019Rigging the Lottery: Making All Tickets Winnerscited by 491
  10. 2020Pruning neural networks without any data by iteratively conserving synaptic flowcited by 475
  11. 2020Picking Winning Tickets Before Training by Preserving Gradient Flowcited by 467
  12. 2024GaLore: Memory-Efficient LLM Training by Gradient Low-Rank Projectioncited by 447
  13. 2024SliceGPT: Compress Large Language Models by Deleting Rows and Columnscited by 378
  14. 2020Visual Transformers: Token-based Image Representation and Processing for Computer Visioncited by 333
  15. 2024Massive Activations in Large Language Modelscited by 277
  16. 2024Extreme Compression of Large Language Models via Additive Quantizationcited by 223
  17. 2021Accelerating Sparse Deep Neural Networkscited by 213
  18. 2021Chasing Sparsity in Vision Transformers: An End-to-End Explorationcited by 181
  19. 2022Structured Pruning Learns Compact and Accurate Modelscited by 167
  20. 2023Outlier Weighed Layerwise Sparsity (OWL): A Missing Secret Sauce for Pruning LLMs to High Sparsitycited by 163
  21. 2021BERT Busters: Outlier Dimensions that Disrupt Transformerscited by 110
  22. 2023LoSparse: Structured Compression of Large Language Models based on Low-Rank and Sparse Approximationcited by 103
  23. 2022Unified Visual Transformer Compressioncited by 79
  24. 2020Accelerating Ill-Conditioned Low-Rank Matrix Estimation via Scaled Gradient Descentcited by 70
  25. 2022Vision Transformer Slimming: Multi-Dimension Searching in Continuous Optimization Spacecited by 66
  26. 2023X-Pruner: eXplainable Pruning for Vision Transformerscited by 66
  27. 2020Progressive Skeletonization: Trimming more fat from a network at initializationcited by 64
  28. 2023Dynamic Sparse No Training: Training-Free Fine-tuning for Sparse LLMscited by 61
  29. 2014Non-convex Robust PCAcited by 41
  30. 2023Fast as CHITA: Neural Network Pruning with Combinatorial Optimizationcited by 40
  31. 2016Fast Algorithms for Robust PCA via Gradient Descentcited by 39
  32. 2024APT: Adaptive Pruning and Tuning Pretrained Language Models for Efficient Training and Inferencecited by 29
  33. 2022Coarsening the Granularity: Towards Structurally Sparse Lottery Ticketscited by 27
  34. 2024Implicit Geometry of Next-token Prediction: From Language Sparsity Patterns to Model Representationscited by 24
  35. 2020Prune Responsiblycited by 20
  36. 2024SLoPe: Double-Pruned Sparse Plus Lazy Low-Rank Adapter Pretraining of LLMscited by 18
  37. 2021Sparse Plus Low Rank Matrix Decomposition: A Discrete Optimization Approachcited by 14
  38. 2024LoRAP: Transformer Sub-Layers Deserve Differentiated Structured Compression for Large Language Modelscited by 12

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 3; 2025: 11; 2026: 5.

19 papers.

Sort:

By area:

  1. 2025Attention Sinks: A 'Catch, Tag, Release' Mechanism for Embeddingscited by 16
  2. 2024SLiM: One-shot Quantization and Sparsity with Low-rank Approximation for LLM Weight Compressioncited by 12
  3. 2024Efficient LLM Inference using Dynamic Input Pruning and Cache-Aware Maskingcited by 10
  4. 2025LOST: Low-rank and Sparse Pre-training for Large Language Modelscited by 10
  5. 2025A3 : an Analytical Low-Rank Approximation Framework for Attentioncited by 7
  6. 2025RainFusion: Adaptive Video Generation Acceleration via Multi-Dimensional Visual Redundancycited by 7
  7. 2024EoRA: Fine-tuning-free Compensation for Compressed LLM with Eigenspace Low-Rank Approximationcited by 6
  8. 2025HASSLE-free: A unified Framework for Sparse plus Low-Rank Matrix Decomposition for LLMscited by 3
  9. 2025TELL-TALE: Task Efficient LLMs with Task Aware Layer Eliminationcited by 2
  10. 2025On Information Geometry and Iterative Optimization in Model Compression: Operator Factorizationcited by 1
  11. 2025Amber Pruner: Leveraging N:M Activation Sparsity for Efficient Prefill in Large Language Modelscited by 1
  12. 2025Sparse corruption in low-rank matrix inference: the PCA benchmarkcited by 1
7 more
  1. 20263BASiL: An Algorithmic Framework for Sparse plus Low-Rank Compression of LLMscited by 1
  2. 2026Supernodes and Halos: Loss-Critical Hubs in LLM Feed-Forward Layerscited by 1
  3. 2026GPTQ-intrinsic LoRA: A Near-optimal Algorithm for Low-precision Quantization with Low-rank Adaptationcited by 1
  4. 2025Accelerating Attention with Basis Decompositioncited by 0
  5. 2025Optimal Singular Damage: Efficient LLM Inference in Low Storage Regimescited by 0
  6. 2026ELAS: Efficient Pre-Training of Low-Rank Large Language Models via 2:4 Activation Sparsitycited by 0
  7. 2026RT-Lynx: Putting GEMM Sparsity in the Right Place for Diffusion Modelscited by 0