mlsanityData to 14 Oct 2026

Advancing Multimodal Large Language Models with Quantization-Aware Scale Learning for Efficient Adaptation

arXiv 2408.03735 ↗First posted 7 Aug 2024cs.CV cs.AI cs.LGcited by 15

On the map: Multimodal LLMs / vision-language models → Visual token pruning and efficiency

Abstract (arXiv)

This paper presents the first study to explore the potential of parameter quantization for multimodal large language models to alleviate the significant resource constraint encountered during vision-language instruction tuning. We introduce a Quantization-aware Scale LeArning method based on multimodal Warmup, termed QSLAW. This method is grounded in two key innovations: (1) The learning of group-wise scale factors for quantized LLM weights to mitigate the quantization error arising from activation outliers and achieve more effective vision-language instruction tuning; (2) The implementation of a multimodal warmup that progressively integrates linguistic and multimodal training samples, thereby preventing overfitting of the quantized model to multimodal data while ensuring stable adaptation of multimodal large language models to downstream vision-language tasks. Extensive experiments demonstrate that models quantized by QSLAW perform on par with, or even surpass, their full-precision counterparts, while facilitating up to 1.4 times reduction in VL tuning time and GPU consumption. Our code is released at https://github.com/xjjxmu/QSLAW.

Key facts. Cited by 15 papers in the map (data through 14 Oct 2026). Placed in Visual token pruning and efficiency, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 20 of its 41 references that are in the map. First posted on arXiv 7 Aug 2024.

Builds on

20 of 41 references.

References · 20

  1. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,599
  2. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 14,132
  3. 2020Measuring Massive Multitask Language Understandingcited by 7,963
  4. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,326
  5. 2018GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understandingcited by 4,899
  6. 2022BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generationcited by 4,322
  7. 2021The Power of Scale for Parameter-Efficient Prompt Tuningcited by 3,795
  8. 2023QLoRA: Efficient Finetuning of Quantized LLMscited by 3,418
  9. 2016Pointer Sentinel Mixture Modelscited by 3,096
  10. 2022Learn to Explain: Multimodal Reasoning via Thought Chains for Science Question Answeringcited by 1,919
  11. 2023Visual ChatGPT: Talking, Drawing and Editing with Visual Foundation Modelscited by 668
  12. 2019Learned Step Size Quantizationcited by 628
8 more
  1. 2021BRECQ: Pushing the Limit of Post-Training Quantization by Block Reconstructioncited by 479
  2. 2024LoRA+: Efficient Low Rank Adaptation of Large Modelscited by 377
  3. 2023LoftQ: LoRA-Fine-Tuning-Aware Quantization for Large Language Modelscited by 196
  4. 2022QDrop: Randomly Dropping Quantization for Extremely Low-bit Post-Training Quantizationcited by 180
  5. 2023Pushing Mixture of Experts to the Limit: Extremely Parameter Efficient MoE for Instruction Tuningcited by 150
  6. 2023Memory-Efficient Fine-Tuning of Compressed Large Language Models via sub-4-bit Integer Quantizationcited by 100
  7. 2021Nonuniform-to-Uniform Quantization: Towards Accurate Quantization via Generalized Straight-Through Estimationcited by 79
  8. 2022NIPQ: Noise proxy-based Integrated Pseudo-Quantizationcited by 35

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 10; 2026: 5.

15 papers.

Sort:

By area:

  1. 2025VLA-Cache: Efficient Vision-Language-Action Manipulation via Adaptive Token Cachingcited by 86
  2. 2025Global Compression Commander: Plug-and-Play Inference Acceleration for High-Resolution Large Vision-Language Modelscited by 32
  3. 2025Reasoning on a Budget: A Survey of Adaptive and Controllable Test-Time Compute in LLMscited by 31
  4. 2025A Survey on Efficient Vision-Language Modelscited by 19
  5. 2025QSVD: Efficient Low-rank Approximation for Unified Query-Key-Value Weight Compression in Low-Precision Vision-Language Modelscited by 16
  6. 2025SalQ-VLM: Fine-Grained Saliency-Guided Quantization for Vision-Language Modelscited by 8
  7. 2026Fine-Grained Post-Training Quantization for Large Vision Language Models with Quantization-Aware Integrated Gradientscited by 8
  8. 2025ToDRE: Effective Visual Token Pruning via Token Diversity and Task Relevancecited by 7
  9. 2026MASQuant: Modality-Aware Smoothing Quantization for Multimodal Large Language Modelscited by 5
  10. 2025ToFu: Visual Tokens Reduction via Fusion for Multi-modal, Multi-patch, Multi-image Taskcited by 3
  11. 2025Revisiting Multimodal KV Cache Compression: A Frequency-Domain-Guided Outlier-KV-Aware Approachcited by 3
  12. 2025Dual Precision Quantization for Efficient and Accurate Deep Neural Networks Inferencecited by 1
3 more
  1. 2026Off-the-shelf Vision Models Benefit Image Manipulation Localizationcited by 0
  2. 2026MorphoQuant: Modality-Aware Quantization for Omni-modal Large Language Modelscited by 0
  3. 2026RGSQ: Riemannian Geometry-Sensitive Quantization for Large Vision-Language Modelscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024Q-VLM: Post-training Quantization for Large Vision-Language Models
  2. 2025Efficiently Integrate Large Language Models with Visual Perception: A Survey from the Training Paradigm Perspective
  3. 2024Enhancing Model Performance: Another Approach to Vision-Language Instruction Tuning
  4. 2026QAPruner: Quantization-Aware Vision Token Pruning for Multimodal Large Language Models
  5. 2023Cheap and Quick: Efficient Vision-Language Instruction Tuning for Large Language Models
  6. 2023QFT: Quantized Full-parameter Tuning of LLMs with Affordable Resources
  7. 2025MQuant: Unleashing the Inference Potential of Multimodal Large Language Models via Full Static Quantization
  8. 2025Test-Time Warmup for Multimodal Large Language Models