HASSLE-free: A unified Framework for Sparse plus Low-Rank Matrix Decomposition for LLMs
Not on the map. Why?
Abstract (arXiv)
The impressive capabilities of large foundation models come at a cost of substantial computing resources to serve them. Compressing these pre-trained models is of practical interest as it can democratize deploying them to the machine learning community at large by lowering the costs associated with inference. A promising compression scheme is to decompose foundation models' dense weights into a sum of sparse plus low-rank matrices. In this paper, we design a unified framework coined HASSLE-free for (semi-structured) sparse plus low-rank matrix decomposition of foundation models. Our framework introduces the local layer-wise reconstruction error objective for this decomposition, we demonstrate that prior work solves a relaxation of this optimization problem; and we provide efficient and scalable methods to minimize the exact introduced optimization problem. HASSLE-free substantially outperforms state-of-the-art methods in terms of the introduced objective and a wide range of LLM evaluation benchmarks. For the Llama3-8B model with a 2:4 sparsity component plus a 64-rank component decomposition, a compression scheme for which recent work shows important inference acceleration on GPUs, HASSLE-free reduces the test perplexity by 12% for the WikiText-2 dataset and reduces the gap (compared to the dense model) of the average of eight popular zero-shot tasks by 15% compared to existing methods.
Key facts. Cited by 3 papers in the map (data through 14 Oct 2026). Builds on 39 of its 59 references that are in the map. First posted on arXiv 2 Feb 2025.
Builds on
39 of 59 references.
References · 39
- 2014Adam: A Method for Stochastic Optimizationcited by 37,240
- 2023GPT-4 Technical Reportcited by 18,396
- 2024The Llama 3 Herd of Modelscited by 14,715
- 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,599
- 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,380
- 2018Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challengecited by 4,886
- 2019HellaSwag: Can a Machine Really Finish Your Sentence?cited by 4,100
- 2022OPT: Open Pre-trained Transformer Language Modelscited by 3,512
- 2016Pointer Sentinel Mixture Modelscited by 3,096
- 2019PIQA: Reasoning about Physical Commonsense in Natural Languagecited by 2,883
- 2023Sparks of Artificial General Intelligence: Early experiments with GPT-4cited by 2,592
- 2019BoolQ: Exploring the Surprising Difficulty of Natural Yes/No Questionscited by 2,183
27 more
- 2015Learning both Weights and Connections for Efficient Neural Networkscited by 2,031
- 2023SparseGPT: Massive Language Models Can Be Accurately Pruned in One-Shotcited by 1,121
- 2019Analyzing Multi-Head Self-Attention: Specialized Heads Do the Heavy Lifting, the Rest Can Be Prunedcited by 856
- 2023A Simple and Effective Pruning Approach for Large Language Modelscited by 800
- 2016Learning Structured Sparsity in Deep Neural Networkscited by 541
- 2016Dynamic Network Surgery for Efficient DNNscited by 233
- 2023LoftQ: LoRA-Fine-Tuning-Aware Quantization for Large Language Modelscited by 196
- 2021Learning N:M Fine-grained Structured Sparse Neural Networks From Scratchcited by 179
- 2022The Optimal BERT Surgeon: Scalable and Accurate Second-Order Pruning for Large Language Modelscited by 115
- 2018Interactive Supercomputing on 40,000 Cores for Machine Learning and Data Analysiscited by 110
- 2023LoSparse: Structured Compression of Large Language Models based on Low-Rank and Sparse Approximationcited by 104
- 2023LQ-LoRA: Low-rank Plus Quantized Matrix Decomposition for Efficient Language Model Finetuningcited by 92
- 2015Fast ConvNets Using Group-wise Brain Damagecited by 82
- 2023Dynamic Sparse No Training: Training-Free Fine-tuning for Sparse LLMscited by 61
- 2021How Well Do Sparse Imagenet Models Transfer?cited by 50
- 2014Non-convex Robust PCAcited by 41
- 2024CATS: Contextually-Aware Thresholding for Sparsity in Large Language Modelscited by 41
- 2023Fast as CHITA: Neural Network Pruning with Combinatorial Optimizationcited by 40
- 2019Careful Selection of Knowledge to solve Open Book Question Answeringcited by 35
- 2022Data-Efficient Structured Pruning via Submodular Optimizationcited by 23
- 2024SLoPe: Double-Pruned Sparse Plus Lazy Low-Rank Adapter Pretraining of LLMscited by 20
- 2024OATS: Outlier-Aware Pruning Through Sparse and Low Rank Decompositioncited by 20
- 2024ALPS: Improved Optimization for Highly Sparse One-Shot Pruning for Large Language Modelscited by 19
- 2024OSSCAR: One-Shot Structured Pruning in Vision and Language Models with Combinatorial Optimizationcited by 18
- 2021Sparse Plus Low Rank Matrix Decomposition: A Discrete Optimization Approachcited by 14
- 2024FALCON: FLOP-Aware Combinatorial Optimization for Neural Network Pruningcited by 11
- 2024Preserving Deep Representations In One-Shot Pruning: A Hessian-Free Second-Order Optimization Frameworkcited by 11
Built on by
Citing papers per year: 2025: 1; 2026: 2.
3 papers.
Similar, not cited
Close in wording, no citation either way
- 2025CoSpaDi: Compressing LLMs via Calibration-Guided Sparse Dictionary Learning
- 2024Compressing Large Language Models using Low Rank and Low Precision Decomposition
- 2024S2FT: Efficient, Scalable and Generalizable LLM Fine-tuning by Structured Sparsity
- 2024Compression for Better: A General and Stable Lossless Compression Framework
- 2024Characterizing the Accuracy -- Efficiency Trade-off of Low-rank Decomposition in Language Models
- 2024Basis Selection: Low-Rank Decomposition of Pretrained Large Language Models for Target Applications
- 2024Data-free Weight Compress and Denoise for Large Language Models
- 2025Assigning Distinct Roles to Quantized and Low-Rank Matrices Toward Optimal Weight Decomposition