mlsanityData to 14 Oct 2026

QA-LoRA: Quantization-Aware Low-Rank Adaptation of Large Language Models

arXiv 2309.14717 ↗First posted 26 Sep 2023cs.LG cs.CLcited by 155

Not on the map. Why?

Abstract (arXiv)

Recently years have witnessed a rapid development of large language models (LLMs). Despite the strong ability in many language-understanding tasks, the heavy computational burden largely restricts the application of LLMs especially when one needs to deploy them onto edge devices. In this paper, we propose a quantization-aware low-rank adaptation (QA-LoRA) algorithm. The motivation lies in the imbalanced degrees of freedom of quantization and adaptation, and the solution is to use group-wise operators which increase the degree of freedom of quantization meanwhile decreasing that of adaptation. QA-LoRA is easily implemented with a few lines of code, and it equips the original LoRA with two-fold abilities: (i) during fine-tuning, the LLM's weights are quantized (e.g., into INT4) to reduce time and memory usage; (ii) after fine-tuning, the LLM and auxiliary weights are naturally integrated into a quantized model without loss of accuracy. We apply QA-LoRA to the LLaMA and LLaMA2 model families and validate its effectiveness in different fine-tuning datasets and downstream scenarios. Code will be made available at https://github.com/yuhuixu1993/qa-lora.

Key facts. Cited by 155 papers in the map (data through 14 Oct 2026). Builds on 45 of its 65 references that are in the map. First posted on arXiv 26 Sep 2023.

Builds on

45 of 65 references.

References · 45

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  2. 2020Language Models are Few-Shot Learnerscited by 32,419
  3. 2023GPT-4 Technical Reportcited by 18,396
  4. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,380
  5. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 14,132
  6. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 13,028
  7. 2020Measuring Massive Multitask Language Understandingcited by 7,963
  8. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,326
  9. 2018Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challengecited by 4,886
  10. 2019Parameter-Efficient Transfer Learning for NLPcited by 4,236
  11. 2019HellaSwag: Can a Machine Really Finish Your Sentence?cited by 4,100
  12. 2021Prefix-Tuning: Optimizing Continuous Prompts for Generationcited by 3,915
33 more
  1. 2022OPT: Open Pre-trained Transformer Language Modelscited by 3,512
  2. 2023QLoRA: Efficient Finetuning of Quantized LLMscited by 3,418
  3. 2019PIQA: Reasoning about Physical Commonsense in Natural Languagecited by 2,883
  4. 2022Self-Instruct: Aligning Language Models with Self-Generated Instructionscited by 2,724
  5. 2023Sparks of Artificial General Intelligence: Early experiments with GPT-4cited by 2,592
  6. 2023A Survey of Large Language Modelscited by 2,576
  7. 2022Emergent Abilities of Large Language Modelscited by 2,512
  8. 2019BoolQ: Exploring the Surprising Difficulty of Natural Yes/No Questionscited by 2,183
  9. 2018Can a Suit of Armor Conduct Electricity? A New Dataset for Open Book Question Answeringcited by 2,164
  10. 2022BLOOM: A 176B-Parameter Open-Access Multilingual Language Modelcited by 1,984
  11. 2022SmoothQuant: Accurate and Efficient Post-Training Quantization for Large Language Modelscited by 1,386
  12. 2023A Survey on Evaluation of Large Language Modelscited by 1,242
  13. 2023SparseGPT: Massive Language Models Can Be Accurately Pruned in One-Shotcited by 1,121
  14. 2022LLM.int8(): 8-bit Matrix Multiplication for Transformers at Scalecited by 906
  15. 2021Towards a Unified View of Parameter-Efficient Transfer Learningcited by 872
  16. 2022GLM-130B: An Open Bilingual Pre-trained Modelcited by 843
  17. 2023A Simple and Effective Pruning Approach for Large Language Modelscited by 800
  18. 2023The Flan Collection: Designing Data and Methods for Effective Instruction Tuningcited by 739
  19. 2021GPT Understands, Toocited by 716
  20. 2023LLM-Pruner: On the Structural Pruning of Large Language Modelscited by 650
  21. 2019Learned Step Size Quantizationcited by 628
  22. 2022ZeroQuant: Efficient and Affordable Post-Training Quantization for Large-Scale Transformerscited by 526
  23. 2023SpQR: A Sparse-Quantized Representation for Near-Lossless LLM Weight Compressioncited by 367
  24. 2019Q-BERT: Hessian Based Ultra Low Precision Quantization of BERTcited by 359
  25. 2023LLM-Adapters: An Adapter Family for Parameter-Efficient Fine-Tuning of Large Language Modelscited by 324
  26. 2023LLM-QAT: Data-Free Quantization Aware Training for Large Language Modelscited by 324
  27. 2023A Survey on Model Compression for Large Language Modelscited by 269
  28. 2022DyLoRA: Parameter Efficient Tuning of Pre-trained Models using Dynamic Search-Free Low-Rank Adaptationcited by 233
  29. 2022Outlier Suppression: Pushing the Limit of Low-bit Transformer Language Modelscited by 181
  30. 2023Solving Challenging Math Word Problems Using GPT-4 Code Interpreter with Code-based Self-Verificationcited by 166
  31. 2023Memory-Efficient Fine-Tuning of Compressed Large Language Models via sub-4-bit Integer Quantizationcited by 100
  32. 2023Stable and low-precision training for large-scale vision-language modelscited by 75
  33. 2023ZeroQuant-FP: A Leap Forward in LLMs Post-Training W4A8 Quantization Using Floating-Point Formatscited by 44

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 4; 2024: 60; 2025: 52; 2026: 39.

155 papers.

Sort:

By area:

  1. 2024Parameter-Efficient Fine-Tuning for Large Models: A Comprehensive Surveycited by 656
  2. 2024PiSSA: Principal Singular Values and Singular Vectors Adaptation of Large Language Modelscited by 269
  3. 2023Parameter-Efficient Fine-Tuning Methods for Pretrained Language Models: A Critical Review and Assessmentcited by 230
  4. 2024A Survey on Efficient Inference for Large Language Modelscited by 205
  5. 2023Efficient Large Language Models: A Surveycited by 186
  6. 2024SVDQuant: Absorbing Outliers by Low-Rank Components for 4-Bit Diffusion Modelscited by 170
  7. 2024EfficientQAT: Efficient Quantization-Aware Training for Large Language Modelscited by 127
  8. 2024A Survey of Resource-efficient LLM and Multimodal Foundation Modelscited by 86
  9. 2024ThinK: Thinner Key Cache by Query-Driven Pruningcited by 68
  10. 2024A Survey on LoRA of Large Language Modelscited by 67
  11. 2025DRIFT: Dynamic Rule-Based Defense with Injection Isolation for Securing LLM Agentscited by 66
  12. 2024Model Compression and Efficient Inference for Large Language Models: A Surveycited by 64
38 more
  1. 2024An empirical study of LLaMA3 quantization: from LLMs to MLLMscited by 64
  2. 2024SliM-LLM: Salience-Driven Mixed-Precision Quantization for Large Language Modelscited by 63
  3. 2024Parameter-Efficient Fine-Tuning in Large Models: A Survey of Methodologiescited by 47
  4. 2024CorDA: Context-Oriented Decomposition Adaptation of Large Language Models for Task-Aware Parameter-Efficient Fine-tuningcited by 44
  5. 2024Low-Rank Quantization-Aware Training for LLMscited by 43
  6. 2024Low-Rank Adaptation for Foundation Models: A Comprehensive Reviewcited by 37
  7. 2024Computational Limits of Low-Rank Adaptation (LoRA) Fine-Tuning for Transformer Modelscited by 31
  8. 2024SpikeLLM: Scaling up Spiking Neural Network to Large Language Models via Saliency-based Spikingcited by 31
  9. 2023NOLA: Compressing LoRA using Linear Combination of Random Basiscited by 29
  10. 2024RSUniVLM: A Unified Vision Language Model for Remote Sensing via Granularity-oriented Mixture of Expertscited by 29
  11. 2024Separable Multi-Concept Erasure from Diffusion Modelscited by 28
  12. 2025How Is LLM Reasoning Distracted by Irrelevant Context? An Analysis Using a Controlled Benchmarkcited by 28
  13. 2024Hyperbolic Fine-Tuning for Large Language Modelscited by 27
  14. 2024Federated LoRA with Sparse Communicationcited by 26
  15. 2024I-LLM: Efficient Integer-Only Inference for Fully-Quantized Low-Bit Large Language Modelscited by 23
  16. 2024A Survey: Collaborative Hardware and Software Design in the Era of Large Language Modelscited by 23
  17. 2025ThinkTrap: Denial-of-Service Attacks against Black-box LLM Services via Infinite Thinkingcited by 23
  18. 2025PoLAR: Polar-Decomposed Low-Rank Adapter Representationcited by 20
  19. 2025LoRI: Reducing Cross-Task Interference in Multi-Task Low-Rank Adaptationcited by 19
  20. 2025A Survey on Efficient Vision-Language Modelscited by 19
  21. 2024Multilingual Large Language Models: A Systematic Surveycited by 18
  22. 2024HQ-DiT: Efficient Diffusion Transformer with FP4 Hybrid Quantizationcited by 17
  23. 2024Empirical Guidelines for Deploying LLMs onto Resource-constrained Edge Devicescited by 17
  24. 2025A Survey of Foundation Model-Powered Recommender Systems: From Feature-Based, Generative to Agentic Paradigmscited by 17
  25. 2025FPTQuant: Function-Preserving Transforms for LLM Quantizationcited by 16
  26. 2024L4Q: Parameter Efficient Quantization-Aware Fine-Tuning on Large Language Modelscited by 15
  27. 2024Bridging The Gap between Low-rank and Orthogonal Adaptation via Householder Reflection Adaptationcited by 15
  28. 2024Zero-shot Generative Large Language Models for Systematic Review Screening Automationcited by 14
  29. 2024Certainly Uncertain: A Benchmark and Metric for Multimodal Epistemic and Aleatoric Awarenesscited by 14
  30. 2024ETHER: Efficient Finetuning of Large-Scale Models with Hyperplane Reflectionscited by 12
  31. 2024Evaluating the Generalization Ability of Quantized LLMs: Benchmark, Analysis, and Toolboxcited by 11
  32. 2024Channel-Wise Mixed-Precision Quantization for Large Language Modelscited by 11
  33. 2024IntLoRA: Integral Low-rank Adaptation of Quantized Diffusion Modelscited by 10
  34. 2025Evaluating Text Creativity across Diverse Domains: A Dataset and Large Language Model Evaluatorcited by 10
  35. 2025Mettle: Meta-Token Learning for Memory-Efficient Audio-Visual Adaptationcited by 10
  36. 2025LOST: Low-rank and Sparse Pre-training for Large Language Modelscited by 10
  37. 2026Learning Rate Matters: Vanilla LoRA May Suffice for LLM Fine-tuningcited by 10
  38. 2024A Fine-tuning Enhanced RAG System with Quantized Influence Measure as AI Judgecited by 9

50 most cited of 155 All 155 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2025CLoQ: Enhancing Fine-Tuning of Quantized LLMs via Calibrated LoRA Initialization
  2. 2024QDyLoRA: Quantized Dynamic Low-Rank Adaptation for Efficient Large Language Model Tuning
  3. 2025DL-QAT: Weight-Decomposed Low-Rank Quantization-Aware Training for Large Language Models
  4. 2023LoftQ: LoRA-Fine-Tuning-Aware Quantization for Large Language Models
  5. 2025LoRA Is Slower Than You Think
  6. 2023INT2.1: Towards Fine-Tunable Quantized Large Language Models with Error Correction through Low-Rank Adaptation
  7. 2023LQ-LoRA: Low-rank Plus Quantized Matrix Decomposition for Efficient Language Model Finetuning
  8. 2026Quantization-Robust LLM Unlearning via Low-Rank Adaptation