mlsanityData to 7 Oct 2026

Q-VLM: Post-training Quantization for Large Vision-Language Models

arXiv 2410.08119 ↗First posted 10 Oct 2024cs.CVcited by 47

On the map: Multimodal LLMs / vision-language models → Visual token pruning and efficiency

Abstract (arXiv)

In this paper, we propose a post-training quantization framework of large vision-language models (LVLMs) for efficient multi-modal inference. Conventional quantization methods sequentially search the layer-wise rounding functions by minimizing activation discretization errors, which fails to acquire optimal quantization strategy without considering cross-layer dependency. On the contrary, we mine the cross-layer dependency that significantly influences discretization errors of the entire vision-language model, and embed this dependency into optimal quantization strategy searching with low search cost. Specifically, we observe the strong correlation between the activation entropy and the cross-layer dependency concerning output discretization errors. Therefore, we employ the entropy as the proxy to partition blocks optimally, which aims to achieve satisfying trade-offs between discretization errors and the search cost. Moreover, we optimize the visual encoder to disentangle the cross-layer dependency for fine-grained decomposition of search space, so that the search cost is further reduced without harming the quantization accuracy. Experimental results demonstrate that our method compresses the memory by 2.78x and increase generate speed by 1.44x about 13B LLaVA model without performance degradation on diverse multi-modal reasoning tasks. Code is available at https://github.com/ChangyuanWang17/QVLM.

Key facts. Cited by 47 papers in the map (data through 7 Oct 2026). Placed in Visual token pruning and efficiency, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 48 of its 54 references that are in the map. First posted on arXiv 10 Oct 2024.

Builds on

48 of 54 references.

References · 48

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  2. 2020Language Models are Few-Shot Learnerscited by 32,199
  3. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  4. 2023Visual Instruction Tuningcited by 7,792
  5. 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,069
  6. 2023Improved Baselines with Visual Instruction Tuningcited by 4,362
  7. 2022BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generationcited by 4,310
  8. 2021Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervisioncited by 3,848
  9. 2017MobileNets: Efficient Convolutional Neural Networks for Mobile Vision Applicationscited by 3,827
  10. 2023QLoRA: Efficient Finetuning of Quantized LLMscited by 3,355
  11. 2016Making the V in VQA Matter: Elevating the Role of Image Understanding in Visual Question Answeringcited by 2,498
  12. 2022Do As I Can, Not As I Say: Grounding Language in Robotic Affordancescited by 2,302
36 more
  1. 2022Learn to Explain: Multimodal Reasoning via Thought Chains for Science Question Answeringcited by 1,881
  2. 2022GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformerscited by 1,838
  3. 2022SmoothQuant: Accurate and Efficient Post-Training Quantization for Large Language Modelscited by 1,338
  4. 2017Vision-and-Language Navigation: Interpreting visually-grounded navigation instructions in real environmentscited by 1,013
  5. 2022LLM.int8(): 8-bit Matrix Multiplication for Transformers at Scalecited by 881
  6. 2018VizWiz Grand Challenge: Answering Visual Questions from Blind Peoplecited by 852
  7. 2023Visual ChatGPT: Talking, Drawing and Editing with Visual Foundation Modelscited by 659
  8. 2019Learned Step Size Quantizationcited by 604
  9. 2020Up or Down? Adaptive Rounding for Post-Training Quantizationcited by 600
  10. 2023OpenFlamingo: An Open-Source Framework for Training Large Autoregressive Vision-Language Modelscited by 560
  11. 2022ZeroQuant: Efficient and Affordable Post-Training Quantization for Large-Scale Transformerscited by 517
  12. 2020The Hateful Memes Challenge: Detecting Hate Speech in Multimodal Memescited by 485
  13. 2018PACT: Parameterized Clipping Activation for Quantized Neural Networkscited by 456
  14. 2024MoE-LLaVA: Mixture of Experts for Large Vision-Language Modelscited by 298
  15. 2021Post-Training Quantization for Vision Transformercited by 274
  16. 2020ZeroQ: A Novel Zero Shot Quantization Frameworkcited by 265
  17. 2019Differentiable Soft Quantization: Bridging Full-Precision and Low-Bit Neural Networkscited by 234
  18. 2020Towards Learning a Generic Agent for Vision-and-Language Navigation via Pre-trainingcited by 228
  19. 2016Network Trimming: A Data-Driven Neuron Pruning Approach towards Efficient Deep Architecturescited by 220
  20. 2019Low-bit Quantization of Neural Networks for Efficient Inferencecited by 185
  21. 2022Outlier Suppression: Pushing the Limit of Low-bit Transformer Language Modelscited by 176
  22. 2024TinyLLaVA: A Framework of Small-scale Large Multimodal Modelscited by 164
  23. 2019Improving Neural Network Quantization without Retraining using Outlier Channel Splittingcited by 150
  24. 2023RPTQ: Reorder-based Post-training Quantization for Large Language Modelscited by 112
  25. 2019Loss Aware Post-training Quantizationcited by 94
  26. 2021Network Quantization with Element-wise Gradient Scalingcited by 82
  27. 2023ZeroQuant-V2: Exploring Post-training Quantization in LLMs from Comprehensive Study to Low Rank Compensationcited by 78
  28. 2023TinyGPT-V: Efficient Multimodal Large Language Model via Small Backbonescited by 77
  29. 2020Group Sparsity: The Hinge Between Filter Pruning and Decomposition for Network Compressioncited by 72
  30. 2022Patch Similarity Aware Data-Free Quantization for Vision Transformerscited by 55
  31. 2023Towards Accurate Post-training Quantization for Diffusion Modelscited by 46
  32. 2018Bi-Real Net: Binarizing Deep Network Towards Real-Network Performancecited by 42
  33. 2019ThunderNet: Towards Real-time Generic Object Detectioncited by 42
  34. 2023CBQ: Cross-Block Quantization for Large Language Modelscited by 34
  35. 2020BiDet: An Efficient Binarized Object Detectorcited by 33
  36. 2021EPE-NAS: Efficient Performance Estimation Without Training for Neural Architecture Searchcited by 24

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 1; 2025: 15; 2026: 31.

47 papers.

Sort:

By area:

  1. 2025LaViDa: A Large Diffusion Language Model for Multimodal Understandingcited by 81
  2. 2026QVLA: Not All Channels Are Equal in Vision-Language-Action Model's Quantizationcited by 23
  3. 2025QSVD: Efficient Low-rank Approximation for Unified Query-Key-Value Weight Compression in Low-Precision Vision-Language Modelscited by 15
  4. 2025Distributed LLMs and Multimodal Large Language Models: A Survey on Advances, Challenges, and Future Directionscited by 10
  5. 2025Speculative Decoding Reimagined for Multimodal Large Language Modelscited by 10
  6. 2024freePruner: A Training-free Approach for Large Multimodal Model Accelerationcited by 7
  7. 2025SalQ-VLM: Fine-Grained Saliency-Guided Quantization for Vision-Language Modelscited by 7
  8. 2026Fine-Grained Post-Training Quantization for Large Vision Language Models with Quantization-Aware Integrated Gradientscited by 7
  9. 2025PLPHP: Per-Layer Per-Head Vision Token Pruning for Efficient Large Vision-Language Modelscited by 6
  10. 2025LatentLLM: Attention-Aware Joint Tensor Compressioncited by 6
  11. 2025Bi-VLM: Pushing Ultra-Low Precision Post-Training Quantization Boundaries in Vision-Language Modelscited by 5
  12. 2026Gated Relational Alignment via Confidence-based Distillation for Efficient VLMscited by 5
35 more
  1. 2026VEQ: Modality-Adaptive Quantization for MoE Vision-Language Modelscited by 4
  2. 2026SAFE-Pruner: Semantic Attention-Guided Future-Aware Token Pruning for Efficient Vision-Language-Action Manipulationcited by 4
  3. 2025μ-MoE: Test-Time Pruning as Micro-Grained Mixture-of-Expertscited by 3
  4. 2025AWP: Activation-Aware Weight Pruning and Quantization with Projected Gradient Descentcited by 3
  5. 2025Less Precise Can Be More Reliable: A Systematic Evaluation of Quantization's Impact on VLMs Beyond Accuracycited by 3
  6. 2026Knowledge Vector Weakening: Efficient Training-free Unlearning for Large Vision-Language Modelscited by 3
  7. 2026WSVD: Weighted Low-Rank Approximation for Fast and Efficient Execution of Low-Precision Vision-Language Modelscited by 3
  8. 2025VLM@school -- Evaluation of AI image understanding on German middle school knowledgecited by 2
  9. 2026Rethinking Practical and Efficient Quantization Calibration for Vision-Language Modelscited by 2
  10. 2026Accelerating Multimodal Large Language Models with Prior-Corrected Token Reductioncited by 2
  11. 2025Dual Precision Quantization for Efficient and Accurate Deep Neural Networks Inferencecited by 1
  12. 2025Scene-Aware Vectorized Memory Multi-Agent Framework with Cross-Modal Differentiated Quantization VLMs for Visually Impaired Assistancecited by 1
  13. 2025Towards Efficient VLMs: Information-Theoretic Driven Compression via Adaptive Structural Pruningcited by 1
  14. 2026Learning Domain Knowledge in Multimodal Large Language Models through Reinforcement Fine-Tuningcited by 1
  15. 2026TTQ: Activation-Aware Test-Time Quantization to Accelerate LLM Inference On The Flycited by 1
  16. 2026QAPruner: Quantization-Aware Vision Token Pruning for Multimodal Large Language Modelscited by 1
  17. 2026DA-PTQ: Drift-Aware Post-Training Quantization for Efficient Vision-Language-Action Modelscited by 1
  18. 2026SparseSAM: Structured Sparsification of Activations in Segment Anything Modelscited by 1
  19. 2026Cut-ViT: Task-Specific Model Pruning via Gram Anchoring Subspace Consistencycited by 1
  20. 2026BEFORE THE FLIP: Measuring Hidden Score Shifts In Quantized Vision Language Models Before The Answer Changes for Visual Question Answeringcited by 1
  21. 2026Fast-Slow Efficient Training for Multimodal Large Language Models via Visual Token Pruningcited by 0
  22. 2026Mostly Text, Smart Visuals: Asymmetric Text-Visual Pruning for Large Vision-Language Modelscited by 0
  23. 2026LLaVA-CKD: Bottom-Up Cascaded Knowledge Distillation for Vision-Language Modelscited by 0
  24. 2026LASER: Loss-Aware Singular-value Decomposition and Rank Allocation for Efficient Low-Precision Vision-Language Modelscited by 0
  25. 2026MorphoQuant: Modality-Aware Quantization for Omni-modal Large Language Modelscited by 0
  26. 2026EinSort: Sorting is All We Need for Tensorizing LLMcited by 0
  27. 2026Token-level Response-visual Attention Guidance for Multimodal LLMs Knowledge Distillationcited by 0
  28. 2026Llama-Mobile: Efficient 2.7-Bit Quantization of VLMscited by 0
  29. 2026Activation Outliers Matter: Robust Recovery for Quantized Multimodal LLMscited by 0
  30. 2026MiX: Micro-Inverted-Scaling for End-to-End Low-Bit Vision-Language Model Accelerationcited by 0
  31. 2026RGSQ: Riemannian Geometry-Sensitive Quantization for Large Vision-Language Modelscited by 0
  32. 2026GHOST-Q: Towards Studying Grounding Hallucinations Overlooked Under Same-score TradeOffs in Quantized VLMScited by 0
  33. 2026Beyond Reconstruction Loss in Post-Training Quantization: Balanced Fitting for Large Vision-Language Modelscited by 0
  34. 2026P4Q: Co-designing Token Pruning and Quantization for Vision-Language Model Accelerationcited by 0
  35. 2026Calibrate the Decisions That Change the Future: On-Policy Post-Training Quantization for Multimodal Large Language Modelscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026Towards Efficient Large Vision-Language Models: A Comprehensive Survey on Inference Strategies
  2. 2026VLAQuantBench: Closed-Loop Evaluation of Post-Training Quantization for Vision-Language-Action Models
  3. 2024Advancing Multimodal Large Language Models with Quantization-Aware Scale Learning for Efficient Adaptation
  4. 2024MBQ: Modality-Balanced Quantization for Large Vision-Language Models
  5. 2026Towards Understanding Best Practices for Quantization of Vision-Language Models
  6. 2026Efficient Inference for Large Vision-Language Models: Bottlenecks, Techniques, and Prospects
  7. 2026Breaking Modality Heterogeneity in Low-Bit Quantization for Large Vision-Language Models
  8. 2022EfficientVLM: Fast and Accurate Vision-Language Models via Knowledge Distillation and Modal-adaptive Pruning