CURing Large Models: Compression via CUR Decomposition
Not on the map. Why?
Abstract (arXiv)
Large deep learning models have achieved remarkable success but are resource-intensive, posing challenges such as memory usage. We introduce CURing, a novel model compression method based on CUR matrix decomposition, which approximates weight matrices as the product of selected columns (C) and rows (R), and a small linking matrix (U). We apply this decomposition to weights chosen based on the combined influence of their magnitudes and activations. By identifying and retaining informative rows and columns, CURing significantly reduces model size with minimal performance loss. For example, it reduces Llama3.1-8B's parameters to 7.32B (-9%) in just 129 seconds, over 20 times faster than prior compression methods.
Key facts. Cited by 1 paper in the map (data through 7 Oct 2026). Builds on 34 of its 64 references that are in the map. First posted on arXiv 8 Jan 2025.
Builds on
34 of 64 references.
References · 34
- 2017Attention Is All You Needcited by 59,932
- 2023GPT-4 Technical Reportcited by 18,132
- 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
- 2024The Llama 3 Herd of Modelscited by 14,192
- 2015Distilling the Knowledge in a Neural Networkcited by 10,076
- 2016SGDR: Stochastic Gradient Descent with Warm Restartscited by 4,993
- 2020GLU Variants Improve Transformercited by 1,724
- 2019Analyzing Multi-Head Self-Attention: Specialized Heads Do the Heavy Lifting, the Rest Can Be Prunedcited by 844
- 2019Are Sixteen Heads Really Better than One?cited by 803
- 2023A Simple and Effective Pruning Approach for Large Language Modelscited by 776
- 2016Language Modeling with Gated Convolutional Networkscited by 688
- 2019Patient Knowledge Distillation for BERT Model Compressioncited by 502
22 more
- 2024SliceGPT: Compress Large Language Models by Deleting Rows and Columnscited by 378
- 2024ShortGPT: Layers in Large Language Models are More Redundant Than You Expectcited by 369
- 2024Better & Faster Large Language Models via Multi-token Predictioncited by 344
- 2024The Unreasonable Ineffectiveness of the Deeper Layerscited by 219
- 2019Structured Pruning of Large Language Modelscited by 201
- 2021Block Pruning For Faster Transformerscited by 176
- 2022Structured Pruning Learns Compact and Accurate Modelscited by 167
- 2024Compact Language Models via Pruning and Knowledge Distillationcited by 142
- 2023The Truth is in There: Improving Reasoning in Language Models with Layer-Selective Rank Reductioncited by 120
- 2021Sparse is Enough in Scaling Transformerscited by 86
- 2020Understanding and Improving Knowledge Distillationcited by 83
- 2024LLM Pruning and Distillation in Practice: The Minitron Approachcited by 82
- 2020LadaBERT: Lightweight Adaptation of BERT through Hybrid Model Compressioncited by 45
- 2024MoRA: High-Rank Updating for Parameter-Efficient Fine-Tuningcited by 31
- 2014Optimal CUR Matrix Decompositionscited by 29
- 2020Solvable Model for Inheriting the Regularization through Knowledge Distillationcited by 15
- 2022Structured Pruning Adapterscited by 12
- 2024Persistent Topological Features in Large Language Modelscited by 11
- 2024CURLoRA: Stable LLM Continual Fine-Tuning and Catastrophic Forgetting Mitigationcited by 10
- 2021Model Preserving Compression for Neural Networkscited by 9
- 2024BitStack: Any-Size Compression of Large Language Models in Variable Memory Environmentscited by 9
- 2024STAT: Shrinking Transformers After Trainingcited by 2
Built on by
Citing papers per year: 2025: 1.
1 paper.