mlsanityData to 7 Oct 2026

LoQT: Low-Rank Adapters for Quantized Pretraining

arXiv 2405.16528 ↗First posted 26 May 2024cs.LG cs.CLcited by 12

Not on the map. Why?

Abstract (arXiv)

Despite advances using low-rank adapters and quantization, pretraining of large models on consumer hardware has not been possible without model sharding, offloading during training, or per-layer gradient updates. To address these limitations, we propose Low-Rank Adapters for Quantized Training (LoQT), a method for efficiently training quantized models. LoQT uses gradient-based tensor factorization to initialize low-rank trainable weight matrices that are periodically merged into quantized full-rank weight matrices. Our approach is suitable for both pretraining and fine-tuning models. We demonstrate this for language modeling and downstream task adaptation, finding that LoQT enables efficient training of models up to 7B parameters on a 24GB GPU. We also demonstrate the feasibility of training a 13B model using per-layer gradient updates on the same hardware.

Key facts. Cited by 12 papers in the map (data through 7 Oct 2026). Builds on 41 of its 45 references that are in the map. First posted on arXiv 26 May 2024.

Builds on

41 of 45 references.

References · 41

  1. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  2. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,084
  3. 2021Training Verifiers to Solve Math Word Problemscited by 9,743
  4. 2018GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understandingcited by 4,854
  5. 2023QLoRA: Efficient Finetuning of Quantized LLMscited by 3,355
  6. 2022Training Compute-Optimal Large Language Modelscited by 2,927
  7. 2022GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformerscited by 1,838
  8. 2022SmoothQuant: Accurate and Efficient Post-Training Quantization for Large Language Modelscited by 1,338
  9. 2021DeBERTaV3: Improving DeBERTa using ELECTRA-Style Pre-Training with Gradient-Disentangled Embedding Sharingcited by 1,219
  10. 2018Adafactor: Adaptive Learning Rates with Sublinear Memory Costcited by 896
  11. 2021A Survey of Quantization Methods for Efficient Neural Network Inferencecited by 798
  12. 2024GaLore: Memory-Efficient LLM Training by Gradient Low-Rank Projectioncited by 447
29 more
  1. 2023OmniQuant: Omnidirectionally Calibrated Quantization for Large Language Modelscited by 403
  2. 20218-bit Optimizers via Block-wise Quantizationcited by 395
  3. 2024LoRA+: Efficient Low Rank Adaptation of Large Modelscited by 365
  4. 2023SpQR: A Sparse-Quantized Representation for Near-Lossless LLM Weight Compressioncited by 359
  5. 2019Q-BERT: Hessian Based Ultra Low Precision Quantization of BERTcited by 355
  6. 2024QuIP#: Even Better LLM Quantization with Hadamard Incoherence and Lattice Codebookscited by 354
  7. 2023LLM-QAT: Data-Free Quantization Aware Training for Large Language Modelscited by 320
  8. 2019Q8BERT: Quantized 8Bit BERTcited by 296
  9. 2024The Era of 1-bit LLMs: All Large Language Models are in 1.58 Bitscited by 289
  10. 2019ERNIE 2.0: A Continual Pre-training Framework for Language Understandingcited by 230
  11. 2024Extreme Compression of Large Language Models via Additive Quantizationcited by 223
  12. 2018Gradient Descent Happens in a Tiny Subspacecited by 208
  13. 2023ReLoRA: High-Rank Training Through Low-Rank Updatescited by 200
  14. 2023LoftQ: LoRA-Fine-Tuning-Aware Quantization for Large Language Modelscited by 195
  15. 2023BitNet: Scaling 1-bit Transformers for Large Language Modelscited by 175
  16. 2018Training Deep Neural Networks with 8-bit Floating Point Numberscited by 163
  17. 2018Learning to Quantize Deep Networks by Optimizing Quantization Intervals with Task Losscited by 157
  18. 2018Scalable Methods for 8-bit Training of Neural Networkscited by 139
  19. 2023Full Parameter Fine-tuning for Large Language Models with Limited Resourcescited by 134
  20. 2022LUT-GEMM: Quantized Matrix Multiplication based on LUTs for Efficient Inference in Large-Scale Generative Language Modelscited by 98
  21. 2023LQ-LoRA: Low-rank Plus Quantized Matrix Decomposition for Efficient Language Model Finetuningcited by 92
  22. 2023Stable and low-precision training for large-scale vision-language modelscited by 74
  23. 2023FlexRound: Learnable Rounding based on Element-wise Division for Post-Training Quantizationcited by 46
  24. 2021Towards Efficient Post-training Quantization of Pre-trained Language Modelscited by 37
  25. 2024Jetfire: Efficient and Accurate Transformer Pretraining with INT8 Data Flow and Per-Block Quantizationcited by 30
  26. 2021How many degrees of freedom do we need to train deep networks: a loss landscape perspectivecited by 28
  27. 2024ApiQ: Finetuning of 2-Bit Quantized Large Language Modelcited by 27
  28. 2021Accurate Neural Training with 4-bit Matrix Multiplications at Standard Formatscited by 26
  29. 2023Training and inference of large language models using 8-bit floating pointcited by 20

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 1; 2025: 7; 2026: 4.

12 papers.

  1. 2024APOLLO: SGD-like Memory, AdamW-level Performancecited by 50
  2. 2025HeadInfer: Memory-Efficient LLM Inference by Head-wise Offloadingcited by 9
  3. 2025TensorGRaD: Tensor Gradient Robust Decomposition for Memory-Efficient Neural Operator Trainingcited by 5
  4. 2025Calibrating and Rotating: A Unified Framework for Weight Conditioning in PEFTcited by 4
  5. 2025BOOST: BOttleneck-Optimized Scalable Training Framework for Low-Rank Large Language Modelscited by 2
  6. 2025Unlocking Full Efficiency of Token Filtering in Large Language Model Trainingcited by 1
  7. 2025On the Duality between Gradient Transformations and Adapterscited by 1
  8. 2025MLoRQ: Bridging Low-Rank and Quantization for Transformer Compressioncited by 1
  9. 2026Train Less, Infer Faster: Efficient Model Finetuning and Compression via Structured Sparsitycited by 1
  10. 2026DecompressionLM: Deterministic, Diagnostic, and Zero-Shot Concept Graph Extraction from Language Modelscited by 0
  11. 2026M+Adam: Low-Precision Training via Additive-Multiplicative Optimizationcited by 0
  12. 2026Fine-Tuning Low-Bit Models with Gradient in Quantized Code Spacecited by 0