mlsanityData to 14 Oct 2026

LoftQ: LoRA-Fine-Tuning-Aware Quantization for Large Language Models

arXiv 2310.08659 ↗First posted 12 Oct 2023cs.CL cs.AI cs.LGcited by 196

Not on the map. Why?

Abstract (arXiv)

Quantization is an indispensable technique for serving Large Language Models (LLMs) and has recently found its way into LoRA fine-tuning. In this work we focus on the scenario where quantization and LoRA fine-tuning are applied together on a pre-trained model. In such cases it is common to observe a consistent gap in the performance on downstream tasks between full fine-tuning and quantization plus LoRA fine-tuning approach. In response, we propose LoftQ (LoRA-Fine-Tuning-aware Quantization), a novel quantization framework that simultaneously quantizes an LLM and finds a proper low-rank initialization for LoRA fine-tuning. Such an initialization alleviates the discrepancy between the quantized and full-precision model and significantly improves generalization in downstream tasks. We evaluate our method on natural language understanding, question answering, summarization, and natural language generation tasks. Experiments show that our method is highly effective and outperforms existing quantization methods, especially in the challenging 2-bit and 2/4-bit mixed precision regimes. The code is available on https://github.com/yxli2123/LoftQ.

Key facts. Cited by 196 papers in the map (data through 14 Oct 2026). Builds on 29 of its 42 references that are in the map. First posted on arXiv 12 Oct 2023.

Builds on

29 of 42 references.

References · 29

  1. 2017Attention Is All You Needcited by 60,361
  2. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,305
  3. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,380
  4. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 13,028
  5. 2021Training Verifiers to Solve Math Word Problemscited by 10,109
  6. 2019BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehensioncited by 5,893
  7. 2018GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understandingcited by 4,899
  8. 2016SQuAD: 100,000+ Questions for Machine Comprehension of Textcited by 4,148
  9. 2023QLoRA: Efficient Finetuning of Quantized LLMscited by 3,418
  10. 2016Pointer Sentinel Mixture Modelscited by 3,096
  11. 2017A Broad-Coverage Challenge Corpus for Sentence Understanding through Inferencecited by 2,682
  12. 2022GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformerscited by 1,919
17 more
  1. 2022SmoothQuant: Accurate and Efficient Post-Training Quantization for Large Language Modelscited by 1,386
  2. 2018Don't Give Me the Details, Just the Summary! Topic-Aware Convolutional Neural Networks for Extreme Summarizationcited by 1,317
  3. 2021DeBERTaV3: Improving DeBERTa using ELECTRA-Style Pre-Training with Gradient-Disentangled Embedding Sharingcited by 1,254
  4. 2015Teaching Machines to Read and Comprehendcited by 1,049
  5. 2017SemEval-2017 Task 1: Semantic Textual Similarity - Multilingual and Cross-lingual Focused Evaluationcited by 1,001
  6. 2018Neural Network Acceptability Judgmentscited by 958
  7. 2022LLM.int8(): 8-bit Matrix Multiplication for Transformers at Scalecited by 906
  8. 2021Towards a Unified View of Parameter-Efficient Transfer Learningcited by 872
  9. 2019Adversarial NLI: A New Benchmark for Natural Language Understandingcited by 807
  10. 2019Q-BERT: Hessian Based Ultra Low Precision Quantization of BERTcited by 359
  11. 2023LLM-QAT: Data-Free Quantization Aware Training for Large Language Modelscited by 324
  12. 2019Q8BERT: Quantized 8Bit BERTcited by 300
  13. 2019Fixup Initialization: Residual Learning Without Normalizationcited by 198
  14. 2020BinaryBERT: Pushing the Limit of BERT Quantizationcited by 178
  15. 2023LoSparse: Structured Compression of Large Language Models based on Low-Rank and Sparse Approximationcited by 104
  16. 2023LMFlow: An Extensible Toolkit for Finetuning and Inference of Large Foundation Modelscited by 60
  17. 2021Towards Efficient Post-training Quantization of Pre-trained Language Modelscited by 39

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 6; 2024: 85; 2025: 66; 2026: 39.

196 papers.

Sort:

By area:

  1. 2024Parameter-Efficient Fine-Tuning for Large Models: A Comprehensive Surveycited by 656
  2. 2024LoRA+: Efficient Low Rank Adaptation of Large Modelscited by 377
  3. 2024LoRA Learns Less and Forgets Lesscited by 328
  4. 2024PiSSA: Principal Singular Values and Singular Vectors Adaptation of Large Language Modelscited by 269
  5. 2023Parameter-Efficient Fine-Tuning Methods for Pretrained Language Models: A Critical Review and Assessmentcited by 230
  6. 2024The Unreasonable Ineffectiveness of the Deeper Layerscited by 226
  7. 2024A Survey on Efficient Inference for Large Language Modelscited by 205
  8. 2023Efficient Large Language Models: A Surveycited by 186
  9. 2024SVDQuant: Absorbing Outliers by Low-Rank Components for 4-Bit Diffusion Modelscited by 170
  10. 2024GEAR: An Efficient KV Cache Compression Recipe for Near-Lossless Generative Inference of LLMcited by 158
  11. 2024LLM Inference Unveiled: Survey and Roofline Model Insightscited by 152
  12. 2024EfficientQAT: Efficient Quantization-Aware Training for Large Language Modelscited by 127
38 more
  1. 2024Vaccine: Perturbation-aware Alignment for Large Language Models against Harmful Fine-tuning Attackcited by 115
  2. 2023LQ-LoRA: Low-rank Plus Quantized Matrix Decomposition for Efficient Language Model Finetuningcited by 92
  3. 2024Booster: Tackling Harmful Fine-tuning for Large Language Models via Attenuating Harmful Perturbationcited by 92
  4. 2024Harmful Fine-tuning Attacks and Defenses for Large Language Models: A Surveycited by 90
  5. 2025A Survey of Graph Retrieval-Augmented Generation for Customized Large Language Modelscited by 88
  6. 2024A Survey of Resource-efficient LLM and Multimodal Foundation Modelscited by 86
  7. 2024GPTVQ: The Blessing of Dimensionality for LLM Quantizationcited by 82
  8. 2024LoRA-XS: Low-Rank Adaptation with Extremely Small Number of Parameterscited by 76
  9. 2024Unveiling and Harnessing Hidden Attention Sinks: Enhancing Large Language Models without Training through Attention Calibrationcited by 74
  10. 2024A Survey on LoRA of Large Language Modelscited by 67
  11. 2024Model Compression and Efficient Inference for Large Language Models: A Surveycited by 64
  12. 2024Compressing Large Language Models using Low Rank and Low Precision Decompositioncited by 59
  13. 2024The Impact of Initialization on LoRA Finetuning Dynamicscited by 54
  14. 2024Multi-Frame, Lightweight & Efficient Vision-Language Models for Question Answering in Autonomous Drivingcited by 52
  15. 2024A Systematic Survey of Text Summarization: From Statistical Methods to Large Language Modelscited by 52
  16. 2024RoSA: Accurate Parameter-Efficient Fine-Tuning via Robust Adaptationcited by 48
  17. 2024Parameter-Efficient Fine-Tuning in Large Models: A Survey of Methodologiescited by 47
  18. 2024CorDA: Context-Oriented Decomposition Adaptation of Large Language Models for Task-Aware Parameter-Efficient Fine-tuningcited by 44
  19. 2024Low-Rank Quantization-Aware Training for LLMscited by 43
  20. 2024LQER: Low-Rank Quantization Error Reconstruction for LLMscited by 41
  21. 2024Accurate LoRA-Finetuning Quantization of LLMs via Information Retentioncited by 41
  22. 2024Low-Rank Adaptation for Foundation Models: A Comprehensive Reviewcited by 37
  23. 2024Computational Limits of Low-Rank Adaptation (LoRA) Fine-Tuning for Transformer Modelscited by 31
  24. 2024Lottery Ticket Adaptation: Mitigating Destructive Interference in LLMscited by 31
  25. 2024A Survey of Low-bit Large Language Models: Basics, Systems, and Algorithmscited by 31
  26. 2024ApiQ: Finetuning of 2-Bit Quantized Large Language Modelcited by 27
  27. 2024Hyperbolic Fine-Tuning for Large Language Modelscited by 27
  28. 2024Quantization-Aware Imitation-Learning for Resource-Efficient Robotic Controlcited by 25
  29. 2024I-LLM: Efficient Integer-Only Inference for Fully-Quantized Low-Bit Large Language Modelscited by 23
  30. 2025Quantization Meets dLLMs: A Systematic Study of Post-training Quantization for Diffusion LLMscited by 23
  31. 2024Efficient Learning With Sine-Activated Low-rank Matricescited by 22
  32. 2024QERA: an Analytical Framework for Quantization Error Reconstructioncited by 22
  33. 2025QVGen: Pushing the Limit of Quantized Video Generative Modelscited by 22
  34. 2024RoseLoRA: Row and Column-wise Sparse Low-rank Adaptation of Pre-trained Language Model for Knowledge Editing and Fine-tuningcited by 21
  35. 2024Comparing Retrieval-Augmentation and Parameter-Efficient Fine-Tuning for Privacy-Preserving Personalization of Large Language Modelscited by 21
  36. 2025GSQ-Tuning: Group-Shared Exponents Integer in Fully Quantized Training for LLMs On-Device Fine-tuningcited by 21
  37. 2024Parameter Efficient Fine-tuning via Explained Variance Adaptationcited by 20
  38. 2025PoLAR: Polar-Decomposed Low-Rank Adapter Representationcited by 20

50 most cited of 196 All 196 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2023QA-LoRA: Quantization-Aware Low-Rank Adaptation of Large Language Models
  2. 2024Bayesian-LoRA: LoRA based Parameter Efficient Fine-Tuning using Optimal Quantization levels and Rank Values trough Differentiable Bayesian Gates
  3. 2024LoRA Land: 310 Fine-tuned LLMs that Rival GPT-4, A Technical Report
  4. 2024LRQ: Optimizing Post-Training Quantization for Large Language Models by Learning Low-Rank Weight-Scaling Matrices
  5. 2023QLLM: Accurate and Efficient Low-Bitwidth Quantization for Large Language Models
  6. 2026SEPTQ: A Simple and Effective Post-Training Quantization Paradigm for Large Language Models
  7. 2023Memory-Efficient Fine-Tuning of Compressed Large Language Models via sub-4-bit Integer Quantization
  8. 2026Learning Rate Matters: Vanilla LoRA May Suffice for LLM Fine-tuning