mlsanityData to 14 Oct 2026

Compressing Large Language Models using Low Rank and Low Precision Decomposition

arXiv 2405.18886 ↗First posted 29 May 2024cs.LG cs.AI math.OCcited by 59

Not on the map. Why?

Abstract (arXiv)

The prohibitive sizes of Large Language Models (LLMs) today make it difficult to deploy them on memory-constrained edge devices. This work introduces rm CALDERA -- a new post-training LLM compression algorithm that harnesses the inherent low-rank structure of a weight matrix W by approximating it via a low-rank, low-precision decomposition as W ≈ Q + LR. Here, L and R are low rank factors, and the entries of Q, L and R are quantized. The model is compressed by substituting each layer with its Q + LR decomposition, and the zero-shot performance of the compressed model is evaluated. Additionally, L and R are readily amenable to low-rank adaptation, consequently enhancing the zero-shot performance. rm CALDERA obtains this decomposition by formulating it as an optimization problem minQ,L,RlVert(Q + LR - W)X^toprVertrm F2, where X is the calibration data, and Q, L, R are constrained to be representable using low-precision formats. Theoretical upper bounds on the approximation error of rm CALDERA are established using a rank-constrained regression framework, and the tradeoff between compression ratio and model performance is studied by analyzing the impact of target rank and quantization bit budget. Results illustrate that compressing LlaMa-2 7B/13B/70B and LlaMa-3 8B models using rm CALDERA outperforms existing post-training LLM compression techniques in the regime of less than 2.5 bits per parameter. The implementation is available at: https://github.com/pilancilab/caldera.

Key facts. Cited by 59 papers in the map (data through 14 Oct 2026). Builds on 19 of its 46 references that are in the map. First posted on arXiv 29 May 2024.

Builds on

19 of 46 references.

References · 19

  1. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,380
  2. 2018GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understandingcited by 4,899
  3. 2018Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challengecited by 4,886
  4. 2023QLoRA: Efficient Finetuning of Quantized LLMscited by 3,418
  5. 2016Pointer Sentinel Mixture Modelscited by 3,096
  6. 2019PIQA: Reasoning about Physical Commonsense in Natural Languagecited by 2,883
  7. 2022SmoothQuant: Accurate and Efficient Post-Training Quantization for Large Language Modelscited by 1,386
  8. 2019ZeRO: Memory Optimizations Toward Training Trillion Parameter Modelscited by 1,301
  9. 2020Up or Down? Adaptive Rounding for Post-Training Quantizationcited by 621
  10. 2021Documenting Large Webtext Corpora: A Case Study on the Colossal Clean Crawled Corpuscited by 492
  11. 2023QuIP: 2-Bit Quantization of Large Language Models With Guaranteescited by 414
  12. 2024The Era of 1-bit LLMs: All Large Language Models are in 1.58 Bitscited by 308
7 more
  1. 2024Extreme Compression of Large Language Models via Additive Quantizationcited by 232
  2. 2023LoftQ: LoRA-Fine-Tuning-Aware Quantization for Large Language Modelscited by 196
  3. 2022Language model compression with weighted low-rank factorizationcited by 195
  4. 2023The Truth is in There: Improving Reasoning in Language Models with Layer-Selective Rank Reductioncited by 123
  5. 2023LQ-LoRA: Low-rank Plus Quantized Matrix Decomposition for Efficient Language Model Finetuningcited by 92
  6. 2023ZeroQuant-V2: Exploring Post-training Quantization in LLMs from Comprehensive Study to Low Rank Compensationcited by 79
  7. 2023Matrix Compression via Randomized Low Rank and Low Precision Factorizationcited by 25

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 6; 2025: 33; 2026: 20.

59 papers.

Sort:

By area:

  1. 2024Small Singular Values Matter: A Random Matrix Analysis of Transformer Modelscited by 21
  2. 2024SLiM: One-shot Quantization and Sparsity with Low-rank Approximation for LLM Weight Compressioncited by 12
  3. 2025TensorLLM: Tensorising Multi-Head Attention for Enhanced Reasoning and Compression in LLMscited by 12
  4. 2025A Survey on Inference Engines for Large Language Models: Perspectives on Optimization and Efficiencycited by 12
  5. 2025LittleBit: Ultra Low-Bit Quantization via Latent Factorizationcited by 11
  6. 2025SALS: Sparse Attention in Latent Space for KV cache Compressioncited by 11
  7. 2026LLaVA-FA: Learning Fourier Approximation for Compressing Large Multimodal Modelscited by 11
  8. 2026Influence-Preserving Proxies for Gradient-Based Data Selection in LLM Fine-tuningcited by 10
  9. 2025A3 : an Analytical Low-Rank Approximation Framework for Attentioncited by 8
  10. 2025Layer-wise dynamic rank for compressing large language modelscited by 8
  11. 2024EoRA: Fine-tuning-free Compensation for Compressed LLM with Eigenspace Low-Rank Approximationcited by 7
  12. 2025LatentLLM: Attention-Aware Joint Tensor Compressioncited by 7
38 more
  1. 2025ERC-SVD: Error-Controlled SVD for Large Language Model Compressioncited by 7
  2. 2024TRAWL: Tensor Reduced and Approximated Weights for Large Language Modelscited by 5
  3. 2025On Accelerating Edge AI: Optimizing Resource-Constrained Environmentscited by 5
  4. 2025FBQuant: FeedBack Quantization for Large Language Modelscited by 5
  5. 2025ImPart: Importance-Aware Delta-Sparsification for Improved Model Compression and Merging in LLMscited by 5
  6. 2025Pivoting Factorization: A Compact Meta Low-Rank Representation of Sparsity for Efficient Inference in Large Language Modelscited by 4
  7. 2025Addition is almost all you need: Compressing large language models with double binary factorizationcited by 3
  8. 2025μ-MoE: Test-Time Pruning as Micro-Grained Mixture-of-Expertscited by 3
  9. 2025ProcrustesGPT: Compressing LLMs with Structured Matrices and Orthogonal Transformationscited by 3
  10. 2025A Comprehensive Evaluation on Quantization Techniques for Large Language Modelscited by 3
  11. 2025PATCH: Learnable Tile-level Hybrid Sparsity for LLMscited by 3
  12. 2025Bandwidth-Efficient Adaptive Mixture-of-Experts via Low-Rank Compensationcited by 3
  13. 2026LoPRo: Enhancing Low-Rank Quantization via Permuted Block-Wise Rotationcited by 3
  14. 20263BASiL: An Algorithmic Framework for Sparse plus Low-Rank Compression of LLMscited by 3
  15. 2024Low-Rank Correction for Quantized LLMscited by 2
  16. 2025Saten: Sparse Augmented Tensor Networks for Post-Training Compression of Large Language Modelscited by 2
  17. 2026FLRQ: Faster LLM Quantization with Flexible Low-Rank Matrix Sketchingcited by 2
  18. 2025NoWag: A Unified Framework for Shape Preserving Compression of Large Language Modelscited by 1
  19. 2025Efficient Large Language Model Inference with Neural Block Linearizationcited by 1
  20. 2025Assigning Distinct Roles to Quantized and Low-Rank Matrices Toward Optimal Weight Decompositioncited by 1
  21. 2025On Information Geometry and Iterative Optimization in Model Compression: Operator Factorizationcited by 1
  22. 2025Entropy Meets Importance: A Unified Head Importance-Entropy Score for Stable and Efficient Transformer Pruningcited by 1
  23. 2026Preserve-Then-Quantize: Balancing Rank Budgets for Quantization Error Reconstruction in LLMscited by 1
  24. 2026Beyond Student: An Asymmetric Network for Neural Network Inheritancecited by 1
  25. 2026SERQ: Saliency-Aware Low-Rank Error Reconstruction for LLM Quantizationcited by 1
  26. 2026TTQ: Activation-Aware Test-Time Quantization to Accelerate LLM Inference On The Flycited by 1
  27. 2026D-QRELO: Training- and Data-Free Delta Compression for Large Language Models via Quantization and Residual Low-Rank Approximationcited by 1
  28. 2026GPTQ-intrinsic LoRA: A Near-optimal Algorithm for Low-precision Quantization with Low-rank Adaptationcited by 1
  29. 2026SAB-LVLM: Significance-Aware Binarization for Large Vision-Language Modelscited by 1
  30. 2024Compensate Quantization Errors+: Quantized Models Are Inquisitive Learnerscited by 0
  31. 2025Payload-Aware Intrusion Detection with CMAE and Large Language Modelscited by 0
  32. 2025Penrose Tiled Low-Rank Compression and Section-Wise Q&A Fine-Tuning: A General Framework for Domain-Specific Large Language Model Adaptationcited by 0
  33. 2025A Review of YOLOv12: Attention-Based Enhancements vs. Previous Versionscited by 0
  34. 2025Exchangeability in Neural Network and its Application to Dynamic Pruningcited by 0
  35. 2025Accelerating Attention with Basis Decompositioncited by 0
  36. 2025Quantization Range Estimation for Convolutional Neural Networkscited by 0
  37. 2025D-com: Accelerating Iterative Processing to Enable Low-rank Decomposition of Activationscited by 0
  38. 2025Binary Quadratic Quantization: Beyond First-Order Quantization for Real-Valued Matrix Compressioncited by 0

50 most cited of 59 All 59 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2026Break Through the Compression Bottleneck: From Theory to Practice
  2. 2024Data-free Weight Compress and Denoise for Large Language Models
  3. 2023Rethinking Compression: Reduced Order Modelling of Latent Features in Large Language Models
  4. 2024Basis Selection: Low-Rank Decomposition of Pretrained Large Language Models for Target Applications
  5. 2025FLAT-LLM: Fine-grained Low-rank Activation Space Transformation for Large Language Model Compression
  6. 2024Characterizing the Accuracy -- Efficiency Trade-off of Low-rank Decomposition in Language Models
  7. 2026Compression Trinity: Exploring Sparsity, Quantization, and Low-Rank Approximations for LLM Compression
  8. 2025PocketLLM: Ultimate Compression of Large Language Models via Meta Networks