mlsanityData to 14 Oct 2026

DL-QAT: Weight-Decomposed Low-Rank Quantization-Aware Training for Large Language Models

arXiv 2504.09223 ↗First posted 12 Apr 2025cs.CV cs.AI cs.LGcited by 9

Not on the map. Why?

Abstract (arXiv)

Improving the efficiency of inference in Large Language Models (LLMs) is a critical area of research. Post-training Quantization (PTQ) is a popular technique, but it often faces challenges at low-bit levels, particularly in downstream tasks. Quantization-aware Training (QAT) can alleviate this problem, but it requires significantly more computational resources. To tackle this, we introduced Weight-Decomposed Low-Rank Quantization-Aware Training (DL-QAT), which merges the advantages of QAT while training only less than 1% of the total parameters. Specifically, we introduce a group-specific quantization magnitude to adjust the overall scale of each quantization group. Within each quantization group, we use LoRA matrices to update the weight size and direction in the quantization space. We validated the effectiveness of our method on the LLaMA and LLaMA2 model families. The results show significant improvements over our baseline method across different quantization granularities. For instance, for LLaMA-7B, our approach outperforms the previous state-of-the-art method by 4.2% in MMLU on 3-bit LLaMA-7B model. Additionally, our quantization results on pre-trained models also surpass previous QAT methods, demonstrating the superior performance and efficiency of our approach.

Key facts. Cited by 9 papers in the map (data through 14 Oct 2026). Builds on 13 of its 17 references that are in the map. First posted on arXiv 12 Apr 2025.

Builds on

13 of 17 references.

References · 13

  1. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,380
  2. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 14,132
  3. 2020Measuring Massive Multitask Language Understandingcited by 7,963
  4. 2023QLoRA: Efficient Finetuning of Quantized LLMscited by 3,418
  5. 2016Pointer Sentinel Mixture Modelscited by 3,096
  6. 2022GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformerscited by 1,919
  7. 2024DoRA: Weight-Decomposed Low-Rank Adaptationcited by 758
  8. 2024QuaRot: Outlier-Free 4-Bit Inference in Rotated LLMscited by 634
  9. 2024LoRA+: Efficient Low Rank Adaptation of Large Modelscited by 377
  10. 2023AdaLoRA: Adaptive Budget Allocation for Parameter-Efficient Fine-Tuningcited by 357
  11. 2023LQ-LoRA: Low-rank Plus Quantized Matrix Decomposition for Efficient Language Model Finetuningcited by 92
  12. 2024Asymmetry in Low-Rank Adapters of Foundation Modelscited by 86
1 more
  1. 2024PeriodicLoRA: Breaking the Low-Rank Bottleneck in LoRA Optimizationcited by 19

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 5; 2026: 4.

9 papers.

  1. 2025LittleBit: Ultra Low-Bit Quantization via Latent Factorizationcited by 11
  2. 2025Enhancing Ultra-Low-Bit Quantization of Large Language Models Through Saliency-Aware Partial Retrainingcited by 3
  3. 2026Quant Experts: Token-aware Adaptive Error Reconstruction with Mixture of Experts for Large Vision-Language Models Quantizationcited by 3
  4. 2025Recover-LoRA: Data-Free Accuracy Recovery of Degraded Language Models via Low-Rank Adaptationcited by 2
  5. 2025MLoRQ: Bridging Low-Rank and Quantization for Transformer Compressioncited by 1
  6. 2026ReQuant: Fixed-Grid Discrete Refinement for Post-Training Quantizationcited by 1
  7. 2025SignRoundV2: Toward Closing the Performance Gap in Extremely Low-Bit Post-Training Quantization for LLMscited by 0
  8. 2026HCInfer: An Efficient Inference System via Error Compensation for Resource-Constrained Devicescited by 0
  9. 2026Recover-LoRA for Aggressive Quantization: Reclaiming Accuracy in 2-Bit Language Models via Low-Rank Adaptation with Knowledge Distillation on Synthetic Datacited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024Low-Rank Quantization-Aware Training for LLMs
  2. 2024LRQ: Optimizing Post-Training Quantization for Large Language Models by Learning Low-Rank Weight-Scaling Matrices
  3. 2023LLM-QAT: Data-Free Quantization Aware Training for Large Language Models
  4. 2024L4Q: Parameter Efficient Quantization-Aware Fine-Tuning on Large Language Models
  5. 2023QA-LoRA: Quantization-Aware Low-Rank Adaptation of Large Language Models
  6. 2026SEPTQ: A Simple and Effective Post-Training Quantization Paradigm for Large Language Models
  7. 2024VPTQ: Extreme Low-bit Vector Post-Training Quantization for Large Language Models
  8. 2025Binary Neural Networks for Large Language Model: A Survey