mlsanityData to 7 Oct 2026

LowRA: Accurate and Efficient LoRA Fine-Tuning of LLMs under 2 Bits

arXiv 2502.08141 ↗First posted 12 Feb 2025cs.LG cs.AR cs.CLcited by 6

Not on the map. Why?

Abstract (arXiv)

Fine-tuning large language models (LLMs) is increasingly costly as models scale to hundreds of billions of parameters, and even parameter-efficient fine-tuning (PEFT) methods like LoRA remain resource-intensive. We introduce LowRA, the first framework to enable LoRA fine-tuning below 2 bits per parameter with minimal performance loss. LowRA optimizes fine-grained quantization - mapping, threshold selection, and precision assignment - while leveraging efficient CUDA kernels for scalable deployment. Extensive evaluations across 4 LLMs and 4 datasets show that LowRA achieves a superior performance-precision trade-off above 2 bits and remains accurate down to 1.15 bits, reducing memory usage by up to 50%. Our results highlight the potential of ultra-low-bit LoRA fine-tuning for resource-constrained environments.

Key facts. Cited by 6 papers in the map (data through 7 Oct 2026). Builds on 37 of its 54 references that are in the map. First posted on arXiv 12 Feb 2025.

Builds on

37 of 54 references.

References · 37

  1. 2017Attention Is All You Needcited by 59,932
  2. 2024The Llama 3 Herd of Modelscited by 14,192
  3. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,084
  4. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  5. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  6. 2019BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehensioncited by 5,874
  7. 2021Finetuned Language Models Are Zero-Shot Learnerscited by 3,747
  8. 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedbackcited by 3,634
  9. 2023QLoRA: Efficient Finetuning of Quantized LLMscited by 3,355
  10. 2016Pointer Sentinel Mixture Modelscited by 2,992
  11. 2019Fine-Tuning Language Models from Human Preferencescited by 2,124
  12. 2018Don't Give Me the Details, Just the Summary! Topic-Aware Convolutional Neural Networks for Extreme Summarizationcited by 1,308
25 more
  1. 2021BitFit: Simple Parameter-efficient Fine-tuning for Transformer-based Masked Language-modelscited by 1,234
  2. 2015Teaching Machines to Read and Comprehendcited by 1,039
  3. 2021A General Language Assistant as a Laboratory for Alignmentcited by 1,023
  4. 2022Few-Shot Parameter-Efficient Fine-Tuning is Better and Cheaper than In-Context Learningcited by 886
  5. 2022Super-NaturalInstructions: Generalization via Declarative Instructions on 1600+ NLP Taskscited by 858
  6. 2023OpenAssistant Conversations -- Democratizing Large Language Model Alignmentcited by 712
  7. 2019Learned Step Size Quantizationcited by 604
  8. 2023Mitigating Hallucination in Large Multi-Modal Models via Robust Instruction Tuningcited by 439
  9. 2019Q-BERT: Hessian Based Ultra Low Precision Quantization of BERTcited by 355
  10. 2024PiSSA: Principal Singular Values and Singular Vectors Adaptation of Large Language Modelscited by 261
  11. 2023Atom: Low-bit Quantization for Efficient and Accurate LLM Servingcited by 255
  12. 2023LoftQ: LoRA-Fine-Tuning-Aware Quantization for Large Language Modelscited by 195
  13. 2020BinaryBERT: Pushing the Limit of BERT Quantizationcited by 175
  14. 2021UniPELT: A Unified Framework for Parameter-Efficient Language Model Tuningcited by 162
  15. 2024SVDQuant: Absorbing Outliers by Low-Rank Components for 4-Bit Diffusion Modelscited by 161
  16. 2024LoRA-GA: Low-Rank Adaptation with Gradient Approximationcited by 134
  17. 2023Punica: Multi-Tenant LoRA Servingcited by 80
  18. 2021Nonuniform-to-Uniform Quantization: Towards Accurate Quantization via Generalized Straight-Through Estimationcited by 78
  19. 2024Towards Modular LLMs by Building and Reusing a Library of LoRAscited by 72
  20. 2024Accurate LoRA-Finetuning Quantization of LLMs via Information Retentioncited by 40
  21. 2021BSQ: Exploring Bit-Level Sparsity for Mixed-Precision Neural Network Quantizationcited by 37
  22. 2024ApiQ: Finetuning of 2-Bit Quantized Large Language Modelcited by 27
  23. 2024Mitigating Large Language Model Hallucination with Faithful Finetuningcited by 24
  24. 2024BitStack: Any-Size Compression of Large Language Models in Variable Memory Environmentscited by 9
  25. 2025FlexQuant: Elastic Quantization Framework for Locally Hosted LLM on Edge Devicescited by 4

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 1; 2026: 5.

6 papers.

  1. 2026ShapLoRA: Allocation of Low-rank Adaption on Large Language Models via Shapley Value Inspired Importance Estimationcited by 3
  2. 2025On-the-Fly Adaptation to Quantization: Configuration-Aware LoRA for Efficient Fine-Tuning of Quantized LLMscited by 1
  3. 2026Sliced-Wasserstein Distribution Alignment Loss Improves the Ultra-Low-Bit Quantization of Large Language Modelscited by 0
  4. 2026Towards a Data-Parameter Correspondence for LLMs: A Preliminary Discussioncited by 0
  5. 2026ProjQ: Project-and-Quantize for Adapter-Aware LLM Compressioncited by 0
  6. 2026AQLoRA: A Zero-Search Recipe for Fast Quantized LoRA Fine-Tuningcited by 0