mlsanityData to 14 Oct 2026

LQ-LoRA: Low-rank Plus Quantized Matrix Decomposition for Efficient Language Model Finetuning

arXiv 2311.12023 ↗First posted 20 Nov 2023cs.CL cs.LGcited by 92

Not on the map. Why?

Abstract (arXiv)

We propose a simple approach for memory-efficient adaptation of pretrained language models. Our approach uses an iterative algorithm to decompose each pretrained matrix into a high-precision low-rank component and a memory-efficient quantized component. During finetuning, the quantized component remains fixed and only the low-rank component is updated. We present an integer linear programming formulation of the quantization component which enables dynamic configuration of quantization parameters (e.g., bit-width, block size) for each matrix given an overall target memory budget. We further explore a data-aware version of the algorithm which uses an approximation of the Fisher information matrix to weight the reconstruction objective during matrix decomposition. Experiments on finetuning RoBERTa and LLaMA-2 (7B and 70B) demonstrate that our low-rank plus quantized matrix decomposition approach (LQ-LoRA) outperforms strong QLoRA and GPTQ-LoRA baselines and enables aggressive quantization to sub-3 bits with only minor performance degradations. When finetuned on a language modeling calibration dataset, LQ-LoRA can also be used for model compression; in this setting our 2.75-bit LLaMA-2-70B model (which has 2.85 bits on average when including the low-rank components and requires 27GB of GPU memory) performs respectably compared to the 16-bit baseline.

Key facts. Cited by 92 papers in the map (data through 14 Oct 2026). Builds on 49 of its 74 references that are in the map. First posted on arXiv 20 Nov 2023.

Builds on

49 of 74 references.

References · 49

  1. 2023GPT-4 Technical Reportcited by 18,396
  2. 2022Training language models to follow instructions with human feedbackcited by 15,971
  3. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,372
  4. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 13,028
  5. 2021Training Verifiers to Solve Math Word Problemscited by 10,109
  6. 2020Measuring Massive Multitask Language Understandingcited by 7,963
  7. 2018GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understandingcited by 4,899
  8. 2018Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challengecited by 4,886
  9. 2019Parameter-Efficient Transfer Learning for NLPcited by 4,236
  10. 2019HellaSwag: Can a Machine Really Finish Your Sentence?cited by 4,100
  11. 2021Prefix-Tuning: Optimizing Continuous Prompts for Generationcited by 3,915
  12. 2021The Power of Scale for Parameter-Efficient Prompt Tuningcited by 3,795
37 more
  1. 2022OPT: Open Pre-trained Transformer Language Modelscited by 3,512
  2. 2023QLoRA: Efficient Finetuning of Quantized LLMscited by 3,418
  3. 2016Pointer Sentinel Mixture Modelscited by 3,096
  4. 2021TruthfulQA: Measuring How Models Mimic Human Falsehoodscited by 2,975
  5. 2022Self-Instruct: Aligning Language Models with Self-Generated Instructionscited by 2,724
  6. 2023Sparks of Artificial General Intelligence: Early experiments with GPT-4cited by 2,592
  7. 2022BLOOM: A 176B-Parameter Open-Access Multilingual Language Modelcited by 1,984
  8. 2022SmoothQuant: Accurate and Efficient Post-Training Quantization for Large Language Modelscited by 1,386
  9. 2021BitFit: Simple Parameter-efficient Fine-tuning for Transformer-based Masked Language-modelscited by 1,245
  10. 2022LLM.int8(): 8-bit Matrix Multiplication for Transformers at Scalecited by 906
  11. 2022ZeroQuant: Efficient and Affordable Post-Training Quantization for Large-Scale Transformerscited by 526
  12. 2023OmniQuant: Omnidirectionally Calibrated Quantization for Large Language Modelscited by 424
  13. 2023SpQR: A Sparse-Quantized Representation for Near-Lossless LLM Weight Compressioncited by 367
  14. 2020Parameter-Efficient Transfer Learning with Diff Pruningcited by 352
  15. 2021ZeRO-Offload: Democratizing Billion-Scale Model Trainingcited by 341
  16. 2022Optimal Brain Compression: A Framework for Accurate Post-Training Quantization and Pruningcited by 322
  17. 2023SqueezeLLM: Dense-and-Sparse Quantizationcited by 294
  18. 2021Parameter-efficient Multi-task Fine-tuning for Transformers via Shared Hypernetworkscited by 284
  19. 2021Training Neural Networks with Fixed Sparse Maskscited by 214
  20. 2023LoftQ: LoRA-Fine-Tuning-Aware Quantization for Large Language Modelscited by 196
  21. 2022Language model compression with weighted low-rank factorizationcited by 195
  22. 2023Memory-Efficient Fine-Tuning of Compressed Large Language Models via sub-4-bit Integer Quantizationcited by 100
  23. 2023ZeroQuant-V2: Exploring Post-training Quantization in LLMs from Comprehensive Study to Low Rank Compensationcited by 79
  24. 2022Mixed-Precision Neural Network Quantization via Learned Layer-wise Importancecited by 54
  25. 2016Fast Algorithms for Robust PCA via Gradient Descentcited by 39
  26. 2021Learned Robust PCA: A Scalable Deep Unfolding Approach for High-Dimensional Outlier Detectioncited by 34
  27. 2023Matrix Compression via Randomized Low Rank and Low Precision Factorizationcited by 25
  28. 2022AlphaTuning: Quantization-Aware Parameter-Efficient Adaptation of Large-Scale Pre-Trained Language Modelscited by 23
  29. 2016Recovery guarantee of weighted low-rank approximation via alternating minimizationcited by 20
  30. 2021KroneckerBERT: Learning Kronecker Decomposition for Pre-trained Language Models via Knowledge Distillationcited by 20
  31. 2023INT2.1: Towards Fine-Tunable Quantized Large Language Models with Error Correction through Low-Rank Adaptationcited by 20
  32. 2017Robust PCA by Manifold Optimizationcited by 13
  33. 2021BMPQ: Bit-Gradient Sensitivity Driven Mixed-Precision Quantization of DNNs from Scratchcited by 11
  34. 2023NF4 Isn't Information Theoretically Optimal (and that's Good)cited by 11
  35. 2014A variational approach to stable principal component pursuitcited by 8
  36. 2018Efficient Optimization Algorithms for Robust Principal Component Analysis and Its Variantscited by 5
  37. 2021Weighted Low-Rank Matrix Approximation: Acceleration and Applicationscited by 2

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 1; 2024: 34; 2025: 38; 2026: 19.

92 papers.

Sort:

By area:

  1. 2024Parameter-Efficient Fine-Tuning for Large Models: A Comprehensive Surveycited by 656
  2. 2024SVDQuant: Absorbing Outliers by Low-Rank Components for 4-Bit Diffusion Modelscited by 170
  3. 2024LLM Inference Unveiled: Survey and Roofline Model Insightscited by 152
  4. 2024EfficientQAT: Efficient Quantization-Aware Training for Large Language Modelscited by 127
  5. 2024Asymmetry in Low-Rank Adapters of Foundation Modelscited by 86
  6. 2024SliM-LLM: Salience-Driven Mixed-Precision Quantization for Large Language Modelscited by 63
  7. 2024PV-Tuning: Beyond Straight-Through Estimation for Extreme LLM Compressioncited by 61
  8. 2024Compressing Large Language Models using Low Rank and Low Precision Decompositioncited by 59
  9. 2024Parameter-Efficient Fine-Tuning in Large Models: A Survey of Methodologiescited by 47
  10. 2024Low-Rank Quantization-Aware Training for LLMscited by 43
  11. 2024Talking Heads: Understanding Inter-layer Communication in Transformer Language Modelscited by 41
  12. 2024Mixture Compressor for Mixture-of-Experts LLMs Gains Morecited by 41
38 more
  1. 2024Low-Rank Adaptation for Foundation Models: A Comprehensive Reviewcited by 37
  2. 2024Computational Limits of Low-Rank Adaptation (LoRA) Fine-Tuning for Transformer Modelscited by 31
  3. 2024Lottery Ticket Adaptation: Mitigating Destructive Interference in LLMscited by 31
  4. 2024A Survey of Low-bit Large Language Models: Basics, Systems, and Algorithmscited by 31
  5. 2024Fast Matrix Multiplications for Lookup Table-Quantized LLMscited by 29
  6. 2024Zeroth-Order Fine-Tuning of LLMs with Extreme Sparsitycited by 28
  7. 2024ApiQ: Finetuning of 2-Bit Quantized Large Language Modelcited by 27
  8. 2024A Survey: Collaborative Hardware and Software Design in the Era of Large Language Modelscited by 23
  9. 2025Quantization Meets dLLMs: A Systematic Study of Post-training Quantization for Diffusion LLMscited by 23
  10. 2024QERA: an Analytical Framework for Quantization Error Reconstructioncited by 22
  11. 2024SLoPe: Double-Pruned Sparse Plus Lazy Low-Rank Adapter Pretraining of LLMscited by 20
  12. 2025LoRI: Reducing Cross-Task Interference in Multi-Task Low-Rank Adaptationcited by 19
  13. 2025QeRL: Beyond Efficiency -- Quantization-enhanced Reinforcement Learning for LLMscited by 16
  14. 2023ZeroQuant(4+2): Redefining LLMs Quantization with a New FP6-Centric Strategy for Diverse Generative Taskscited by 14
  15. 2024Agentic Retrieval-Augmented Generation for Time Series Analysiscited by 14
  16. 2024LoQT: Low-Rank Adapters for Quantized Pretrainingcited by 12
  17. 2024SLiM: One-shot Quantization and Sparsity with Low-rank Approximation for LLM Weight Compressioncited by 12
  18. 2025UORA: Uniform Orthogonal Reinitialization Adaptation in Parameter-Efficient Fine-Tuning of Large Modelscited by 11
  19. 2025Sharp Generalization Bounds for Foundation Models with Asymmetric Randomized Low-Rank Adapterscited by 11
  20. 2026LLaVA-FA: Learning Fourier Approximation for Compressing Large Multimodal Modelscited by 11
  21. 2024Collage: Light-Weight Low-Precision Strategy for LLM Trainingcited by 10
  22. 2025Efficient Ensemble for Fine-tuning Language Models on Multiple Datasetscited by 10
  23. 2024A Fine-tuning Enhanced RAG System with Quantized Influence Measure as AI Judgecited by 9
  24. 2024Cherry on Top: Parameter Heterogeneity and Quantization in Large Language Modelscited by 9
  25. 2024RILQ: Rank-Insensitive LoRA-based Quantization Error Compensation for Boosting 2-bit Large Language Model Accuracycited by 9
  26. 2025DL-QAT: Weight-Decomposed Low-Rank Quantization-Aware Training for Large Language Modelscited by 9
  27. 2025Quantized but Deceptive? A Multi-Dimensional Truthfulness Evaluation of Quantized LLMscited by 9
  28. 2024ASER: Activation Smoothing and Error Reconstruction for Large Language Model Quantizationcited by 8
  29. 2025Towards Superior Quantization Accuracy: A Layer-sensitive Approachcited by 8
  30. 2025Quaff: Quantized Parameter-Efficient Fine-Tuning under Outlier Spatial Stability Hypothesiscited by 7
  31. 2025CLoQ: Enhancing Fine-Tuning of Quantized LLMs via Calibrated LoRA Initializationcited by 5
  32. 2025Train Small, Infer Large: Memory-Efficient LoRA Training for Large Language Modelscited by 5
  33. 2025CASP: Compression of Large Multimodal Models Based on Attention Sparsitycited by 5
  34. 2025Balancing Fidelity and Plasticity: Aligning Mixed-Precision Fine-Tuning with Linguistic Hierarchiescited by 5
  35. 2025Exploring Layer-wise Information Effectiveness for Post-Training Quantization in Small Language Modelscited by 5
  36. 2026AutoQRA: Joint Optimization of Mixed-Precision Quantization and Low-rank Adapters for Efficient LLM Fine-Tuningcited by 5
  37. 2025Dynamic Context-oriented Decomposition for Task-aware Low-rank Adaptation with Less Forgetting and Faster Convergencecited by 4
  38. 2025LoRA-DA: Data-Aware Initialization for Low-Rank Adaptation via Asymptotic Analysiscited by 4

50 most cited of 92 All 92 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2024L4Q: Parameter Efficient Quantization-Aware Fine-Tuning on Large Language Models
  2. 2023QA-LoRA: Quantization-Aware Low-Rank Adaptation of Large Language Models
  3. 2024LQER: Low-Rank Quantization Error Reconstruction for LLMs
  4. 2025LoRAQuant: Mixed-Precision Quantization of LoRA to Ultra-Low Bits
  5. 2024Accurate and Efficient Fine-Tuning of Quantized Large Language Models Through Optimal Balance
  6. 2024LRQ: Optimizing Post-Training Quantization for Large Language Models by Learning Low-Rank Weight-Scaling Matrices
  7. 2024DoTA: Weight-Decomposed Tensor Adaptation for Large Language Models
  8. 2024QDyLoRA: Quantized Dynamic Low-Rank Adaptation for Efficient Large Language Model Tuning