ClusComp: A Simple Paradigm for Model Compression and Efficient Finetuning
Not on the map. Why?
Abstract (arXiv)
As large language models (LLMs) scale, model compression is crucial for edge deployment and accessibility. Weight-only quantization reduces model size but suffers from performance degradation at lower bit widths. Moreover, standard finetuning is incompatible with quantized models, and alternative methods often fall short of full finetuning. In this paper, we propose ClusComp, a simple yet effective compression paradigm that clusters weight matrices into codebooks and finetunes them block-by-block. ClusComp (1) achieves superior performance in 2-4 bit quantization, (2) pushes compression to 1-bit while outperforming ultra-low-bit methods with minimal finetuning, and (3) enables efficient finetuning, even surpassing existing quantization-based approaches and rivaling full FP16 finetuning. Notably, ClusComp supports compression and finetuning of 70B LLMs on a single A6000-48GB GPU.
Key facts. Cited by 1 paper in the map (data through 7 Oct 2026). Builds on 44 of its 61 references that are in the map. First posted on arXiv 17 Mar 2025.
Builds on
44 of 61 references.
References · 44
- 2017Attention Is All You Needcited by 59,932
- 2014Adam: A Method for Stochastic Optimizationcited by 36,960
- 2020Language Models are Few-Shot Learnerscited by 32,199
- 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
- 2024The Llama 3 Herd of Modelscited by 14,192
- 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
- 2021Training Verifiers to Solve Math Word Problemscited by 9,743
- 2020Measuring Massive Multitask Language Understandingcited by 7,768
- 2018Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challengecited by 4,710
- 2019HellaSwag: Can a Machine Really Finish Your Sentence?cited by 4,018
- 2022OPT: Open Pre-trained Transformer Language Modelscited by 3,464
- 2023QLoRA: Efficient Finetuning of Quantized LLMscited by 3,355
32 more
- 2016Pointer Sentinel Mixture Modelscited by 2,992
- 2019PIQA: Reasoning about Physical Commonsense in Natural Languagecited by 2,818
- 2019BoolQ: Exploring the Surprising Difficulty of Natural Yes/No Questionscited by 2,140
- 2023MMBench: Is Your Multi-modal Model an All-around Player?cited by 2,105
- 2013Estimating or Propagating Gradients Through Stochastic Neurons for Conditional Computationcited by 2,031
- 2022BLOOM: A 176B-Parameter Open-Access Multilingual Language Modelcited by 1,968
- 2022GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformerscited by 1,838
- 2022ChartQA: A Benchmark for Question Answering about Charts with Visual and Logical Reasoningcited by 1,449
- 2020DocVQA: A Dataset for VQA on Document Imagescited by 1,288
- 2023AWQ: Activation-aware Weight Quantization for LLM Compression and Accelerationcited by 1,234
- 2016A Diagram Is Worth A Dozen Imagescited by 793
- 2024Autoregressive Model Beats Diffusion: Llama for Scalable Image Generationcited by 760
- 2023AGIEval: A Human-Centric Benchmark for Evaluating Foundation Modelscited by 701
- 2024LLaVA-NeXT-Interleave: Tackling Multi-image, Video, and 3D in Large Multimodal Modelscited by 571
- 2024GaLore: Memory-Efficient LLM Training by Gradient Low-Rank Projectioncited by 447
- 2023QuIP: 2-Bit Quantization of Large Language Models With Guaranteescited by 397
- 2023SpQR: A Sparse-Quantized Representation for Near-Lossless LLM Weight Compressioncited by 359
- 2024QuIP#: Even Better LLM Quantization with Hadamard Incoherence and Lattice Codebookscited by 354
- 2024LoRA Learns Less and Forgets Lesscited by 313
- 2024Extreme Compression of Large Language Models via Additive Quantizationcited by 223
- 2023Quantizable Transformers: Removing Outliers by Helping Attention Heads Do Nothingcited by 147
- 2024BiLLM: Pushing the Limit of Post-Training Quantization for LLMscited by 137
- 2023Memory-Efficient Fine-Tuning of Compressed Large Language Models via sub-4-bit Integer Quantizationcited by 97
- 2024LISA: Layerwise Importance Sampling for Memory-Efficient Large Language Model Fine-Tuningcited by 88
- 2024GPTVQ: The Blessing of Dimensionality for LLM Quantizationcited by 78
- 2024An empirical study of LLaMA3 quantization: from LLMs to MLLMscited by 63
- 2023Parameter-Efficient Fine-Tuning without Introducing New Latencycited by 59
- 2024PV-Tuning: Beyond Straight-Through Estimation for Extreme LLM Compressioncited by 59
- 2024Any-Precision LLM: Low-Cost Deployment of Multiple, Different-Sized LLMscited by 50
- 2024DB-LLM: Accurate Dual-Binarization for Efficient LLMscited by 35
- 2024ApiQ: Finetuning of 2-Bit Quantized Large Language Modelcited by 27
- 20243-in-1: 2D Rotary Adaptation for Efficient Finetuning, Efficient Batching and Composabilitycited by 10
Built on by
Citing papers per year: 2025: 1.
1 paper.