mlsanityData to 14 Oct 2026

AQLoRA: A Zero-Search Recipe for Fast Quantized LoRA Fine-Tuning

arXiv 2608.23816 ↗First posted 24 Aug 2026cs.LGcited by 0

Not on the map. Why?

Abstract (arXiv)

Quantized fine-tuning (QLoRA) saves memory but not time. It dequantizes every 4-bit weight on the fly, so it trains more slowly than fp16 LoRA. We present AQLoRA (Adaptive-Quantization LoRA), a recipe that buys part of that time back. One CPU pass ranks layers by NF4 reconstruction error, with no search and no calibration data, and keeps a fixed fraction of them in fp16. Those layers skip dequantization, which is where the speed comes from. A quality setting adapts every layer. A speed setting adapts only the top blocks, so the backward pass stops early. The selection matches the accuracy of Unsloth's hand-curated dynamic-4bit list. We evaluate on Commonsense-170K across six models and four architecture families, from 1.4B to 14B. The speed setting trains 11.3 +/- 2.3% faster than well-tuned QLoRA and gives up 0.1 to 1.5 accuracy points. It was faster in all thirteen independent timing sessions, at worst by 7%. The quality setting trains 4.8 +/- 2.0% faster, matches QLoRA's accuracy on every model, and is within a point of fp16 LoRA. The memory premium depends on the architecture: 0.06 to 0.8 GiB on the 1-4B models, 0.5 GiB at 7B, 1.9 GiB at 14B. These error bars are measured between independent sessions, not within one. Timing on shared hardware needs three rules: fix the measurement duration, not the step count; measure the noise floor from a duplicated arm; and repeat whole sessions, since on one model three sessions that each passed the floor test spread over 8.2 points. We validate the recipe with controls and report the two that failed. Choosing adapter layers by weight density is no better than random. Choosing protected layers by quantization error is not either, on five models at three budgets with memory-matched controls and downstream on two. The number of protected layers sets the speed; which layers they are does not measurably change accuracy.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 40 of its 42 references that are in the map. First posted on arXiv 24 Aug 2026.

Builds on

40 of 42 references.

References · 40

  1. 2023QLoRA: Efficient Finetuning of Quantized LLMscited by 3,418
  2. 2024DoRA: Weight-Decomposed Low-Rank Adaptationcited by 758
  3. 2023LLM-Adapters: An Adapter Family for Parameter-Efficient Fine-Tuning of Large Language Modelscited by 324
  4. 2024PiSSA: Principal Singular Values and Singular Vectors Adaptation of Large Language Modelscited by 269
  5. 2019HAWQ-V2: Hessian Aware trace-Weighted Quantization of Neural Networkscited by 261
  6. 2023A Rank Stabilization Scaling Factor for Fine-Tuning with LoRAcited by 243
  7. 2022LST: Ladder Side-Tuning for Parameter and Memory Efficient Transfer Learningcited by 230
  8. 2020AdapterDrop: On the Efficiency of Adapters in Transformerscited by 224
  9. 2023LoftQ: LoRA-Fine-Tuning-Aware Quantization for Large Language Modelscited by 196
  10. 2020Integer Quantization for Deep Learning Inference: Principles and Empirical Evaluationcited by 188
  11. 2023QA-LoRA: Quantization-Aware Low-Rank Adaptation of Large Language Modelscited by 155
  12. 2023OWQ: Outlier-Aware Weight Quantization for Efficient Fine-Tuning and Inference of Large Language Modelscited by 114
28 more
  1. 2023LQ-LoRA: Low-rank Plus Quantized Matrix Decomposition for Efficient Language Model Finetuningcited by 92
  2. 2023Parameter-Efficient Fine-Tuning without Introducing New Latencycited by 59
  3. 2024Accurate LoRA-Finetuning Quantization of LLMs via Information Retentioncited by 41
  4. 2024LoRA-drop: Efficient LoRA Parameter Pruning based on Output Evaluationcited by 38
  5. 2024ApiQ: Finetuning of 2-Bit Quantized Large Language Modelcited by 27
  6. 2024EDGE-LLM: Enabling Efficient Large Language Model Adaptation on Edge Devices via Layerwise Unified Compression and Adaptive Layer Tuning and Votingcited by 27
  7. 2023PockEngine: Sparse and Efficient Fine-tuning in a Pocketcited by 23
  8. 2022Egeria: Efficient DNN Training with Knowledge-Guided Layer Freezingcited by 22
  9. 2024QERA: an Analytical Framework for Quantization Error Reconstructioncited by 22
  10. 2024Layer-Wise Quantization: A Pragmatic and Effective Method for Quantizing LLMs Beyond Integer Bit-Levelscited by 16
  11. 2025QeRL: Beyond Efficiency -- Quantization-enhanced Reinforcement Learning for LLMscited by 16
  12. 2023Towards Green AI in Fine-tuning Large Language Models via Adaptive Backpropagationcited by 12
  13. 2025SparseLoRA: Accelerating LLM Fine-Tuning with Contextual Sparsitycited by 9
  14. 2024DropBP: Accelerating Fine-Tuning of Large Language Models by Dropping Backward Propagationcited by 7
  15. 2024QEFT: Quantization for Efficient Fine-Tuning of LLMscited by 7
  16. 2023ModuLoRA: Finetuning 2-Bit LLMs on Consumer GPUs by Integrating with Modular Quantizerscited by 6
  17. 2025LowRA: Accurate and Efficient LoRA Fine-Tuning of LLMs under 2 Bitscited by 6
  18. 2025CLoQ: Enhancing Fine-Tuning of Quantized LLMs via Calibrated LoRA Initializationcited by 5
  19. 2025Balancing Fidelity and Plasticity: Aligning Mixed-Precision Fine-Tuning with Linguistic Hierarchiescited by 5
  20. 2026AutoQRA: Joint Optimization of Mixed-Precision Quantization and Low-rank Adapters for Efficient LLM Fine-Tuningcited by 5
  21. 2025Beyond Low-Rank Tuning: Model Prior-Guided Rank Allocation for Effective Transfer in Low-Data and Large-Gap Regimescited by 3
  22. 2025LoRA Is Slower Than You Thinkcited by 3
  23. 2025End-to-End On-Device Quantization-Aware Training for LLMs at Inference Costcited by 3
  24. 2026Aletheia: Gradient-Guided Layer Selection for Efficient LoRA Fine-Tuning Across Architecturescited by 3
  25. 2025MobileFineTuner: A Mobile-Native Framework for On-Device LLM Fine-Tuning in Real-World Embedded AI Applicationscited by 2
  26. 2025Memory-Efficient Backpropagation for Fine-Tuning LLMs on Resource-Constrained Mobile Devicescited by 1
  27. 2025FALQON: Accelerating LoRA Fine-tuning with Low-Bit Floating-Point Arithmeticcited by 1
  28. 2026LCSB: Layer-Cyclic Selective Backpropagation for Memory-Efficient On-Device LLM Fine-Tuningcited by 1

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2026Techniques for Peak Memory Reduction for LoRA Fine-tuning of LLMs on Edge Devices
  2. 2025On-the-Fly Adaptation to Quantization: Configuration-Aware LoRA for Efficient Fine-Tuning of Quantized LLMs
  3. 2024QuAILoRA: Quantization-Aware Initialization for LoRA
  4. 2026RAMP: Reinforcement Adaptive Mixed Precision Quantization for Efficient On Device LLM Inference
  5. 2024QDyLoRA: Quantized Dynamic Low-Rank Adaptation for Efficient Large Language Model Tuning
  6. 2026Fine-Tuning Low-Bit Models with Gradient in Quantized Code Space
  7. 2026Task-Aware QUBO Allocation for Mixed-Precision Quantization
  8. 2025JAQ: Joint Efficient Architecture Design and Low-Bit Quantization with Hardware-Software Co-Exploration