mlsanityData to 14 Oct 2026

QLoRA: Efficient Finetuning of Quantized LLMs

arXiv 2305.14314 ↗First posted 23 May 2023cs.LGcited by 3,418

Not on the map. Why?

Cited by 170 in LLM reasoning, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

We present QLoRA, an efficient finetuning approach that reduces memory usage enough to finetune a 65B parameter model on a single 48GB GPU while preserving full 16-bit finetuning task performance. QLoRA backpropagates gradients through a frozen, 4-bit quantized pretrained language model into Low Rank Adapters~(LoRA). Our best model family, which we name Guanaco, outperforms all previous openly released models on the Vicuna benchmark, reaching 99.3% of the performance level of ChatGPT while only requiring 24 hours of finetuning on a single GPU. QLoRA introduces a number of innovations to save memory without sacrificing performance: (a) 4-bit NormalFloat (NF4), a new data type that is information theoretically optimal for normally distributed weights (b) double quantization to reduce the average memory footprint by quantizing the quantization constants, and (c) paged optimziers to manage memory spikes. We use QLoRA to finetune more than 1,000 models, providing a detailed analysis of instruction following and chatbot performance across 8 instruction datasets, multiple model types (LLaMA, T5), and model scales that would be infeasible to run with regular finetuning (e.g. 33B and 65B parameter models). Our results show that QLoRA finetuning on a small high-quality dataset leads to state-of-the-art results, even when using smaller models than the previous SoTA. We provide a detailed analysis of chatbot performance based on both human and GPT-4 evaluations showing that GPT-4 evaluations are a cheap and reasonable alternative to human evaluation. Furthermore, we find that current chatbot benchmarks are not trustworthy to accurately evaluate the performance levels of chatbots. A lemon-picked analysis demonstrates where Guanaco fails compared to ChatGPT. We release all of our models and code, including CUDA kernels for 4-bit training.

Key facts. Cited by 3,418 papers in the map (data through 14 Oct 2026). Builds on 61 of its 73 references that are in the map. First posted on arXiv 23 May 2023.

Builds on

61 of 73 references.

References · 61

  1. 2023GPT-4 Technical Reportcited by 18,396
  2. 2022Training language models to follow instructions with human feedbackcited by 15,971
  3. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,599
  4. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,380
  5. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 14,132
  6. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,372
  7. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,459
  8. 2020Measuring Massive Multitask Language Understandingcited by 7,963
  9. 2018GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understandingcited by 4,899
  10. 2019Parameter-Efficient Transfer Learning for NLPcited by 4,236
  11. 2021On the Opportunities and Risks of Foundation Modelscited by 4,033
  12. 2021Prefix-Tuning: Optimizing Continuous Prompts for Generationcited by 3,915
38 more
  1. 2017Deep reinforcement learning from human preferencescited by 3,912
  2. 2021The Power of Scale for Parameter-Efficient Prompt Tuningcited by 3,795
  3. 2021Finetuned Language Models Are Zero-Shot Learnerscited by 3,780
  4. 2018HotpotQA: A Dataset for Diverse, Explainable Multi-hop Question Answeringcited by 3,754
  5. 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedbackcited by 3,720
  6. 2022OPT: Open Pre-trained Transformer Language Modelscited by 3,512
  7. 2022Constitutional AI: Harmlessness from AI Feedbackcited by 3,050
  8. 2022Scaling Instruction-Finetuned Language Modelscited by 3,024
  9. 2019The Curious Case of Neural Text Degenerationcited by 2,916
  10. 2022Self-Instruct: Aligning Language Models with Self-Generated Instructionscited by 2,724
  11. 2019HuggingFace's Transformers: State-of-the-art Natural Language Processingcited by 2,218
  12. 2022BLOOM: A 176B-Parameter Open-Access Multilingual Language Modelcited by 1,984
  13. 2022GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformerscited by 1,919
  14. 2023Pythia: A Suite for Analyzing Large Language Models Across Training and Scalingcited by 1,786
  15. 2021Multitask Prompted Training Enables Zero-Shot Task Generalizationcited by 1,470
  16. 2022SmoothQuant: Accurate and Efficient Post-Training Quantization for Large Language Modelscited by 1,386
  17. 2021BitFit: Simple Parameter-efficient Fine-tuning for Transformer-based Masked Language-modelscited by 1,245
  18. 2022LaMDA: Language Models for Dialog Applicationscited by 1,220
  19. 2021A General Language Assistant as a Laboratory for Alignmentcited by 1,055
  20. 2022LLM.int8(): 8-bit Matrix Multiplication for Transformers at Scalecited by 906
  21. 2022Few-Shot Parameter-Efficient Fine-Tuning is Better and Cheaper than In-Context Learningcited by 896
  22. 2022Super-NaturalInstructions: Generalization via Declarative Instructions on 1600+ NLP Taskscited by 868
  23. 2016Training Deep Nets with Sublinear Memory Costcited by 848
  24. 2022GLM-130B: An Open Bilingual Pre-trained Modelcited by 843
  25. 2023The Flan Collection: Designing Data and Methods for Effective Instruction Tuningcited by 739
  26. 2023OpenAssistant Conversations -- Democratizing Large Language Model Alignmentcited by 721
  27. 2023Instruction Tuning with GPT-4cited by 686
  28. 2018Annotation Artifacts in Natural Language Inference Datacited by 671
  29. 2022Efficiently Scaling Transformer Inferencecited by 539
  30. 2022Improving alignment of dialogue agents via targeted human judgementscited by 531
  31. 2022ZeroQuant: Efficient and Affordable Post-Training Quantization for Large-Scale Transformerscited by 526
  32. 2021Compacter: Efficient Low-Rank Hypercomplex Adapter Layerscited by 460
  33. 2021MetaICL: Learning to Learn In Contextcited by 454
  34. 20218-bit Optimizers via Block-wise Quantizationcited by 403
  35. 2022Unnatural Instructions: Tuning Language Models with (Almost) No Human Laborcited by 397
  36. 2021Learning How to Ask: Querying LMs with Mixtures of Soft Promptscited by 377
  37. 2023GPTScore: Evaluate as You Desirecited by 361
  38. 2018Hypothesis Only Baselines in Natural Language Inferencecited by 295

50 most cited of 61 All 61 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 314; 2024: 1173; 2025: 934; 2026: 997.

3,418 papers.

Sort:

By area:

  1. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,626
  2. 2023Qwen Technical Reportcited by 3,344
  3. 2023Universal and Transferable Adversarial Attacks on Aligned Language Modelscited by 3,104
  4. 2024OpenVLA: An Open-Source Vision-Language-Action Modelcited by 2,887
  5. 2024LlamaFactory: Unified Efficient Fine-Tuning of 100+ Language Modelscited by 1,464
  6. 2024Medusa: Simple LLM Inference Acceleration Framework with Multiple Decoding Headscited by 843
  7. 2023AutoDAN: Generating Stealthy Jailbreak Prompts on Aligned Large Language Modelscited by 785
  8. 2024DoRA: Weight-Decomposed Low-Rank Adaptationcited by 758
  9. 2023MetaMath: Bootstrap Your Own Mathematical Questions for Large Language Modelscited by 735
  10. 2023WizardCoder: Empowering Code Large Language Models with Evol-Instructcited by 733
  11. 2023Time-LLM: Time Series Forecasting by Reprogramming Large Language Modelscited by 700
  12. 2024A Survey on Large Language Models for Code Generationcited by 658
38 more
  1. 2024Parameter-Efficient Fine-Tuning for Large Models: A Comprehensive Surveycited by 656
  2. 2023Baseline Defenses for Adversarial Attacks Against Aligned Language Modelscited by 651
  3. 2023A Comprehensive Overview of Large Language Modelscited by 648
  4. 2023WizardMath: Empowering Mathematical Reasoning for Large Language Models via Reinforced Evol-Instructcited by 625
  5. 2023Instruction Tuning for Large Language Models: A Surveycited by 623
  6. 2024KVQuant: Towards 10 Million Context Length LLM Inference with KV Cache Quantizationcited by 591
  7. 2024ORPO: Monolithic Preference Optimization without Reference Modelcited by 562
  8. 2024Large Language Models: A Surveycited by 525
  9. 2023MAmmoTH: Building Math Generalist Models through Hybrid Instruction Tuningcited by 487
  10. 2024GaLore: Memory-Efficient LLM Training by Gradient Low-Rank Projectioncited by 463
  11. 2023OmniQuant: Omnidirectionally Calibrated Quantization for Large Language Modelscited by 424
  12. 2023Sheared LLaMA: Accelerating Language Model Pre-training via Structured Pruningcited by 415
  13. 2023How Far Can Camels Go? Exploring the State of Instruction Tuning on Open Resourcescited by 408
  14. 2023Safety-Tuned LLaMAs: Lessons From Improving the Safety of Large Language Models that Follow Instructionscited by 387
  15. 2024ShortGPT: Layers in Large Language Models are More Redundant Than You Expectcited by 383
  16. 2024LoRA+: Efficient Low Rank Adaptation of Large Modelscited by 377
  17. 2023Making Retrieval-Augmented Language Models Robust to Irrelevant Contextcited by 369
  18. 2023Large Language Models for Information Retrieval: A Surveycited by 331
  19. 2024LoRA Learns Less and Forgets Lesscited by 328
  20. 2023SeeSR: Towards Semantics-Aware Real-World Image Super-Resolutioncited by 315
  21. 2024BioMistral: A Collection of Open-Source Pretrained Large Language Models for Medical Domainscited by 301
  22. 2023VeRA: Vector-based Random Matrix Adaptationcited by 300
  23. 2023SqueezeLLM: Dense-and-Sparse Quantizationcited by 294
  24. 2024A Survey on Knowledge Distillation of Large Language Modelscited by 284
  25. 2024MM-LLMs: Recent Advances in MultiModal Large Language Modelscited by 278
  26. 2023MEDITRON-70B: Scaling Medical Pretraining for Large Language Modelscited by 274
  27. 2025Small Language Models are the Future of Agentic AIcited by 270
  28. 2023A Survey on Model Compression for Large Language Modelscited by 269
  29. 2024PiSSA: Principal Singular Values and Singular Vectors Adaptation of Large Language Modelscited by 269
  30. 2023Benchmarking and Defending Against Indirect Prompt Injection Attacks on Large Language Modelscited by 264
  31. 2023FireAct: Toward Language Agent Fine-tuningcited by 256
  32. 2023Challenges and Applications of Large Language Modelscited by 254
  33. 2023FinGPT: Open-Source Financial Large Language Modelscited by 253
  34. 2023OpenChat: Advancing Open-source Language Models with Mixed-Quality Datacited by 252
  35. 2023Multimodal Foundation Models: From Specialists to General-Purpose Assistantscited by 247
  36. 2024SafeDecoding: Defending against Jailbreak Attacks via Safety-Aware Decodingcited by 247
  37. 2023Adapted Large Language Models Can Outperform Medical Experts in Clinical Text Summarizationcited by 246
  38. 2023Large Language Model Alignment: A Surveycited by 243

50 most cited of 3,418 All 3,418 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2023ModuLoRA: Finetuning 2-Bit LLMs on Consumer GPUs by Integrating with Modular Quantizers
  2. 2024LLMEasyQuant: Scalable Quantization for Parallel and Distributed LLM Inference
  3. 2023FineQuant: Unlocking Efficiency with Fine-Grained Weight-Only Quantization for LLMs
  4. 2025GANQ: GPU-Adaptive Non-Uniform Quantization for Large Language Models
  5. 2026TurboBoA: Faster and Exact Attention-aware Quantization without Backpropagation
  6. 2024Spectra: Surprising Effectiveness of Pretraining Ternary Language Models at Scale
  7. 2025Systematic Characterization of LLM Quantization: A Performance, Energy, and Quality Perspective
  8. 2024LoRA Land: 310 Fine-tuned LLMs that Rival GPT-4, A Technical Report