mlsanityData to 14 Oct 2026

QEFT: Quantization for Efficient Fine-Tuning of LLMs

arXiv 2410.08661 ↗First posted 11 Oct 2024cs.CL cs.LGcited by 7

Not on the map. Why?

Abstract (arXiv)

With the rapid growth in the use of fine-tuning for large language models (LLMs), optimizing fine-tuning while keeping inference efficient has become highly important. However, this is a challenging task as it requires improvements in all aspects, including inference speed, fine-tuning speed, memory consumption, and, most importantly, model quality. Previous studies have attempted to achieve this by combining quantization with fine-tuning, but they have failed to enhance all four aspects simultaneously. In this study, we propose a new lightweight technique called Quantization for Efficient Fine-Tuning (QEFT). QEFT accelerates both inference and fine-tuning, is supported by robust theoretical foundations, offers high flexibility, and maintains good hardware compatibility. Our extensive experiments demonstrate that QEFT matches the quality and versatility of full-precision parameter-efficient fine-tuning, while using fewer resources. Our code is available at https://github.com/xvyaward/qeft.

Key facts. Cited by 7 papers in the map (data through 14 Oct 2026). Builds on 28 of its 43 references that are in the map. First posted on arXiv 11 Oct 2024.

Builds on

28 of 43 references.

References · 28

  1. 2020Language Models are Few-Shot Learnerscited by 32,419
  2. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,305
  3. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 14,132
  4. 2021Training Verifiers to Solve Math Word Problemscited by 10,109
  5. 2020Measuring Massive Multitask Language Understandingcited by 7,963
  6. 2018Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challengecited by 4,886
  7. 2019Parameter-Efficient Transfer Learning for NLPcited by 4,236
  8. 2019HellaSwag: Can a Machine Really Finish Your Sentence?cited by 4,100
  9. 2022OPT: Open Pre-trained Transformer Language Modelscited by 3,512
  10. 2023QLoRA: Efficient Finetuning of Quantized LLMscited by 3,418
  11. 2021TruthfulQA: Measuring How Models Mimic Human Falsehoodscited by 2,975
  12. 2023ToolLLM: Facilitating Large Language Models to Master 16000+ Real-world APIscited by 1,947
16 more
  1. 2022Fast Inference from Transformers via Speculative Decodingcited by 1,653
  2. 2021BitFit: Simple Parameter-efficient Fine-tuning for Transformer-based Masked Language-modelscited by 1,245
  3. 2023SparseGPT: Massive Language Models Can Be Accurately Pruned in One-Shotcited by 1,121
  4. 2022Few-Shot Parameter-Efficient Fine-Tuning is Better and Cheaper than In-Context Learningcited by 896
  5. 2019Learned Step Size Quantizationcited by 628
  6. 2024KVQuant: Towards 10 Million Context Length LLM Inference with KV Cache Quantizationcited by 591
  7. 2022ZeroQuant: Efficient and Affordable Post-Training Quantization for Large-Scale Transformerscited by 526
  8. 2021BRECQ: Pushing the Limit of Post-Training Quantization by Block Reconstructioncited by 479
  9. 2023ToolkenGPT: Augmenting Frozen Language Models with Massive Tools via Tool Embeddingscited by 240
  10. 2022QDrop: Randomly Dropping Quantization for Extremely Low-bit Post-Training Quantizationcited by 180
  11. 2022On the Effectiveness of Parameter-Efficient Fine-Tuningcited by 141
  12. 2019Loss Aware Post-training Quantizationcited by 97
  13. 2023Hydra: Multi-head Low-rank Adaptation for Parameter Efficient Fine-tuningcited by 17
  14. 2023ZeroQuant(4+2): Redefining LLMs Quantization with a New FP6-Centric Strategy for Diverse Generative Taskscited by 14
  15. 2022QReg: On Regularization Effects of Quantizationcited by 11
  16. 2023TEQ: Trainable Equivalent Transformation for Quantization of LLMscited by 9

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 1; 2025: 4; 2026: 2.

7 papers.

  1. 2025AMQ: Enabling AutoML for Mixed-precision Weight-Only Quantization of Large Language Modelscited by 14
  2. 2025FlowerTune: A Cross-Domain Benchmark for Federated Fine-Tuning of Large Language Modelscited by 11
  3. 2024PTQ4VM: Post-Training Quantization for Visual Mambacited by 10
  4. 2025Quaff: Quantized Parameter-Efficient Fine-Tuning under Outlier Spatial Stability Hypothesiscited by 7
  5. 2025Adaptive and Resource-efficient Agentic AI Systems for Mobile and Embedded Devices: A Surveycited by 2
  6. 2026AQLoRA: A Zero-Search Recipe for Fast Quantized LoRA Fine-Tuningcited by 0
  7. 2026A Target-Centric Survey of Quantization-Aware Trainingcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2023QFT: Quantized Full-parameter Tuning of LLMs with Affordable Resources
  2. 2024Quantized Side Tuning: Fast and Memory-Efficient Tuning of Quantized Large Language Models
  3. 2024Evaluating the Generalization Ability of Quantized LLMs: Benchmark, Analysis, and Toolbox
  4. 2023Memory-Efficient Fine-Tuning of Compressed Large Language Models via sub-4-bit Integer Quantization
  5. 2026QTALE: Quantization-Robust Token-Adaptive Layer Execution for LLMs
  6. 2025QLESS: A Quantized Approach for Data Valuation and Selection in Large Language Model Fine-Tuning
  7. 2024"Give Me BF16 or Give Me Death"? Accuracy-Performance Trade-Offs in LLM Quantization
  8. 2025FlexQuant: A Flexible and Efficient Dynamic Precision Switching Framework for LLM Quantization