# Compressing Large Language Models using Low Rank and Low Precision Decomposition

arXiv: https://arxiv.org/abs/2405.18886 · HTML: https://mlsanity.com/paper/2405.18886/

## Key facts

- Cited by 59 papers in the map (data through 14 Oct 2026).
- Builds on 19 of its 46 references that are in the map.
- First posted on arXiv 29 May 2024.

## Builds on (most cited first, 19 of 19)

- [LoRA: Low-Rank Adaptation of Large Language Models](https://mlsanity.com/paper/2106.09685/) (2021)
- [GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understanding](https://mlsanity.com/paper/1804.07461/) (2018)
- [Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challenge](https://mlsanity.com/paper/1803.05457/) (2018)
- [QLoRA: Efficient Finetuning of Quantized LLMs](https://mlsanity.com/paper/2305.14314/) (2023)
- [Pointer Sentinel Mixture Models](https://mlsanity.com/paper/1609.07843/) (2016)
- [PIQA: Reasoning about Physical Commonsense in Natural Language](https://mlsanity.com/paper/1911.11641/) (2019)
- [SmoothQuant: Accurate and Efficient Post-Training Quantization for Large Language Models](https://mlsanity.com/paper/2211.10438/) (2022)
- [ZeRO: Memory Optimizations Toward Training Trillion Parameter Models](https://mlsanity.com/paper/1910.02054/) (2019)
- [Up or Down? Adaptive Rounding for Post-Training Quantization](https://mlsanity.com/paper/2004.10568/) (2020)
- [Documenting Large Webtext Corpora: A Case Study on the Colossal Clean Crawled Corpus](https://mlsanity.com/paper/2104.08758/) (2021)
- [QuIP: 2-Bit Quantization of Large Language Models With Guarantees](https://mlsanity.com/paper/2307.13304/) (2023)
- [The Era of 1-bit LLMs: All Large Language Models are in 1.58 Bits](https://mlsanity.com/paper/2402.17764/) (2024)
- [Extreme Compression of Large Language Models via Additive Quantization](https://mlsanity.com/paper/2401.06118/) (2024)
- [LoftQ: LoRA-Fine-Tuning-Aware Quantization for Large Language Models](https://mlsanity.com/paper/2310.08659/) (2023)
- [Language model compression with weighted low-rank factorization](https://mlsanity.com/paper/2207.00112/) (2022)
- [The Truth is in There: Improving Reasoning in Language Models with Layer-Selective Rank Reduction](https://mlsanity.com/paper/2312.13558/) (2023)
- [LQ-LoRA: Low-rank Plus Quantized Matrix Decomposition for Efficient Language Model Finetuning](https://mlsanity.com/paper/2311.12023/) (2023)
- [ZeroQuant-V2: Exploring Post-training Quantization in LLMs from Comprehensive Study to Low Rank Compensation](https://mlsanity.com/paper/2303.08302/) (2023)
- [Matrix Compression via Randomized Low Rank and Low Precision Factorization](https://mlsanity.com/paper/2310.11028/) (2023)

## Built on by (most cited first, 30 of 59)

- [Small Singular Values Matter: A Random Matrix Analysis of Transformer Models](https://mlsanity.com/paper/2410.17770/) (2024)
- [SLiM: One-shot Quantization and Sparsity with Low-rank Approximation for LLM Weight Compression](https://mlsanity.com/paper/2410.09615/) (2024)
- [TensorLLM: Tensorising Multi-Head Attention for Enhanced Reasoning and Compression in LLMs](https://mlsanity.com/paper/2501.15674/) (2025)
- [A Survey on Inference Engines for Large Language Models: Perspectives on Optimization and Efficiency](https://mlsanity.com/paper/2505.01658/) (2025)
- [LittleBit: Ultra Low-Bit Quantization via Latent Factorization](https://mlsanity.com/paper/2506.13771/) (2025)
- [SALS: Sparse Attention in Latent Space for KV cache Compression](https://mlsanity.com/paper/2510.24273/) (2025)
- [LLaVA-FA: Learning Fourier Approximation for Compressing Large Multimodal Models](https://mlsanity.com/paper/2602.00135/) (2026)
- [Influence-Preserving Proxies for Gradient-Based Data Selection in LLM Fine-tuning](https://mlsanity.com/paper/2602.17835/) (2026)
- [A3 : an Analytical Low-Rank Approximation Framework for Attention](https://mlsanity.com/paper/2505.12942/) (2025)
- [Layer-wise dynamic rank for compressing large language models](https://mlsanity.com/paper/2509.25622/) (2025)
- [EoRA: Fine-tuning-free Compensation for Compressed LLM with Eigenspace Low-Rank Approximation](https://mlsanity.com/paper/2410.21271/) (2024)
- [LatentLLM: Attention-Aware Joint Tensor Compression](https://mlsanity.com/paper/2505.18413/) (2025)
- [ERC-SVD: Error-Controlled SVD for Large Language Model Compression](https://mlsanity.com/paper/2505.20112/) (2025)
- [TRAWL: Tensor Reduced and Approximated Weights for Large Language Models](https://mlsanity.com/paper/2406.17261/) (2024)
- [On Accelerating Edge AI: Optimizing Resource-Constrained Environments](https://mlsanity.com/paper/2501.15014/) (2025)
- [FBQuant: FeedBack Quantization for Large Language Models](https://mlsanity.com/paper/2501.16385/) (2025)
- [ImPart: Importance-Aware Delta-Sparsification for Improved Model Compression and Merging in LLMs](https://mlsanity.com/paper/2504.13237/) (2025)
- [Pivoting Factorization: A Compact Meta Low-Rank Representation of Sparsity for Efficient Inference in Large Language Models](https://mlsanity.com/paper/2501.19090/) (2025)
- [Addition is almost all you need: Compressing large language models with double binary factorization](https://mlsanity.com/paper/2505.11076/) (2025)
- [μ-MoE: Test-Time Pruning as Micro-Grained Mixture-of-Experts](https://mlsanity.com/paper/2505.18451/) (2025)
- [ProcrustesGPT: Compressing LLMs with Structured Matrices and Orthogonal Transformations](https://mlsanity.com/paper/2506.02818/) (2025)
- [A Comprehensive Evaluation on Quantization Techniques for Large Language Models](https://mlsanity.com/paper/2507.17417/) (2025)
- [PATCH: Learnable Tile-level Hybrid Sparsity for LLMs](https://mlsanity.com/paper/2509.23410/) (2025)
- [Bandwidth-Efficient Adaptive Mixture-of-Experts via Low-Rank Compensation](https://mlsanity.com/paper/2512.17073/) (2025)
- [LoPRo: Enhancing Low-Rank Quantization via Permuted Block-Wise Rotation](https://mlsanity.com/paper/2601.19675/) (2026)
- [3BASiL: An Algorithmic Framework for Sparse plus Low-Rank Compression of LLMs](https://mlsanity.com/paper/2603.01376/) (2026)
- [Low-Rank Correction for Quantized LLMs](https://mlsanity.com/paper/2412.07902/) (2024)
- [Saten: Sparse Augmented Tensor Networks for Post-Training Compression of Large Language Models](https://mlsanity.com/paper/2505.14871/) (2025)
- [FLRQ: Faster LLM Quantization with Flexible Low-Rank Matrix Sketching](https://mlsanity.com/paper/2601.05684/) (2026)
- [NoWag: A Unified Framework for Shape Preserving Compression of Large Language Models](https://mlsanity.com/paper/2504.14569/) (2025)

## Abstract (arXiv)

The prohibitive sizes of Large Language Models (LLMs) today make it difficult to deploy them on memory-constrained edge devices. This work introduces rm CALDERA -- a new post-training LLM compression algorithm that harnesses the inherent low-rank structure of a weight matrix W by approximating it via a low-rank, low-precision decomposition as W ≈ Q + LR. Here, L and R are low rank factors, and the entries of Q, L and R are quantized. The model is compressed by substituting each layer with its Q + LR decomposition, and the zero-shot performance of the compressed model is evaluated. Additionally, L and R are readily amenable to low-rank adaptation, consequently enhancing the zero-shot performance. rm CALDERA obtains this decomposition by formulating it as an optimization problem minQ,L,RlVert(Q + LR - W)X^toprVertrm F2, where X is the calibration data, and Q, L, R are constrained to be representable using low-precision formats. Theoretical upper bounds on the approximation error of rm CALDERA are established using a rank-constrained regression framework, and the tradeoff between compression ratio and model performance is studied by analyzing the impact of target rank and quantization bit budget. Results illustrate that compressing LlaMa-2 7B/13B/70B and LlaMa-3 8B models using rm CALDERA outperforms existing post-training LLM compression techniques in the regime of less than 2.5 bits per parameter. The implementation is available at: https://github.com/pilancilab/caldera.