mlsanityData to 7 Oct 2026

MBQ: Modality-Balanced Quantization for Large Vision-Language Models

arXiv 2412.19509 ↗First posted 27 Dec 2024cs.CV cs.AIcited by 45

On the map: Multimodal LLMs / vision-language models → Visual token pruning and efficiency

Abstract (arXiv)

Vision-Language Models (VLMs) have enabled a variety of real-world applications. The large parameter size of VLMs brings large memory and computation overhead which poses significant challenges for deployment. Post-Training Quantization (PTQ) is an effective technique to reduce the memory and computation overhead. Existing PTQ methods mainly focus on large language models (LLMs), without considering the differences across other modalities. In this paper, we discover that there is a significant difference in sensitivity between language and vision tokens in large VLMs. Therefore, treating tokens from different modalities equally, as in existing PTQ methods, may over-emphasize the insensitive modalities, leading to significant accuracy loss. To deal with the above issue, we propose a simple yet effective method, Modality-Balanced Quantization (MBQ), for large VLMs. Specifically, MBQ incorporates the different sensitivities across modalities during the calibration process to minimize the reconstruction loss for better quantization parameters. Extensive experiments show that MBQ can significantly improve task accuracy by up to 4.4% and 11.6% under W3 and W4A8 quantization for 7B to 70B VLMs, compared to SOTA baselines. Additionally, we implement a W3 GPU kernel that fuses the dequantization and GEMV operators, achieving a 1.4x speedup on LLaVA-onevision-7B on the RTX 4090. The code is available at https://github.com/thu-nics/MBQ.

Key facts. Cited by 45 papers in the map (data through 7 Oct 2026). Placed in Visual token pruning and efficiency, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 50 of its 58 references that are in the map. First posted on arXiv 27 Dec 2024.

Builds on

50 of 58 references.

Closest ancestors · 4

Same direction, newest first

  1. 2024Boosting Multimodal Large Language Models with Visual Tokens Withdrawal for Rapid Inference9 May
  2. 2024LLaVA-PruMerge: Adaptive Token Reduction for Efficient Large Multimodal Models22 Mar
  3. 2024LLaVA-UHD: an LMM Perceiving Any Aspect Ratio and High-Resolution Images18 Mar
  4. 2024An Image is Worth 1/2 Tokens After Layer 2: Plug-and-Play Inference Acceleration for Large Vision-Language Models11 Mar

Other references · 46

  1. 2017Attention Is All You Needcited by 59,932
  2. 2023Visual Instruction Tuningcited by 7,792
  3. 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,069
  4. 2022Flamingo: a Visual Language Model for Few-Shot Learningcited by 4,337
  5. 2023Sigmoid Loss for Language Image Pre-Trainingcited by 3,048
  6. 2024LLaVA-OneVision: Easy Visual Task Transfercited by 2,669
  7. 2023FlashAttention-2: Faster Attention with Better Parallelism and Work Partitioningcited by 2,551
  8. 2023InternVL: Scaling up Vision Foundation Models and Aligning for Generic Visual-Linguistic Taskscited by 2,534
  9. 2015VQA: Visual Question Answeringcited by 2,489
  10. 2020The Pile: An 800GB Dataset of Diverse Text for Language Modelingcited by 2,346
  11. 2023Efficient Streaming Language Models with Attention Sinkscited by 2,115
  12. 2023MMMU: A Massive Multi-discipline Multimodal Understanding and Reasoning Benchmark for Expert AGIcited by 2,044
34 more
  1. 2022Learn to Explain: Multimodal Reasoning via Thought Chains for Science Question Answeringcited by 1,881
  2. 2022GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformerscited by 1,838
  3. 2019Towards VQA Models That Can Readcited by 1,696
  4. 2015Microsoft COCO Captions: Data Collection and Evaluation Servercited by 1,447
  5. 2022SmoothQuant: Accurate and Efficient Post-Training Quantization for Large Language Modelscited by 1,338
  6. 2023AWQ: Activation-aware Weight Quantization for LLM Compression and Accelerationcited by 1,234
  7. 2023ShareGPT4V: Improving Large Multi-Modal Models with Better Captionscited by 1,000
  8. 2023SEED-Bench: Benchmarking Multimodal LLMs with Generative Comprehensioncited by 929
  9. 2018VizWiz Grand Challenge: Answering Visual Questions from Blind Peoplecited by 852
  10. 2023H2O: Heavy-Hitter Oracle for Efficient Generative Inference of Large Language Modelscited by 805
  11. 2023VILA: On Pre-training for Visual Language Modelscited by 686
  12. 2024PaliGemma: A versatile 3B VLM for transfercited by 679
  13. 2024QuaRot: Outlier-Free 4-Bit Inference in Rotated LLMscited by 598
  14. 2024EAGLE: Speculative Sampling Requires Rethinking Feature Uncertaintycited by 564
  15. 2023OCRBench: On the Hidden Mystery of OCR in Large Multimodal Modelscited by 466
  16. 2024SpinQuant: LLM quantization with learned rotationscited by 446
  17. 2023OmniQuant: Omnidirectionally Calibrated Quantization for Large Language Modelscited by 403
  18. 2023QuIP: 2-Bit Quantization of Large Language Models With Guaranteescited by 397
  19. 2023SpQR: A Sparse-Quantized Representation for Near-Lossless LLM Weight Compressioncited by 359
  20. 2024LMMs-Eval: Reality Check on the Evaluation of Large Multimodal Modelscited by 344
  21. 2024MoE-LLaVA: Mixture of Experts for Large Vision-Language Modelscited by 298
  22. 2023Atom: Low-bit Quantization for Efficient and Accurate LLM Servingcited by 255
  23. 2024MM1: Methods, Analysis & Insights from Multimodal LLM Pre-trainingcited by 230
  24. 2024QServe: W4A8KV4 Quantization and System Co-design for Efficient LLM Servingcited by 200
  25. 2024A Survey on Efficient Inference for Large Language Modelscited by 197
  26. 2024LLaVA-Phi: Efficient Multi-Modal Assistant with Small Language Modelcited by 128
  27. 2024FlatQuant: Flatness Matters for LLM Quantizationcited by 116
  28. 2023Skeleton-of-Thought: Prompting LLMs for Efficient Parallel Generationcited by 103
  29. 2024Cobra: Extending Mamba to Multi-Modal Large Language Model for Efficient Inferencecited by 97
  30. 2024VL-Mamba: Exploring State Space Models for Multimodal Learningcited by 93
  31. 2023ZeroQuant-V2: Exploring Post-training Quantization in LLMs from Comprehensive Study to Low Rank Compensationcited by 78
  32. 2024Evaluating Quantized Large Language Modelscited by 75
  33. 2024MADTP: Multimodal Alignment-Guided Dynamic Token Pruning for Accelerating Vision-Language Transformercited by 53
  34. 2024Mixture of Attention Spans: Optimizing LLM Inference Efficiency with Heterogeneous Sliding-Window Lengthscited by 44

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 9; 2026: 36.

45 papers.

Sort:

By area:

  1. 2025MQuant: Unleashing the Inference Potential of Multimodal Large Language Models via Full Static Quantizationcited by 35
  2. 2025PAROAttention: Pattern-Aware ReOrdering for Efficient Sparse and Quantized Attention in Visual Generation Modelscited by 17
  3. 2025QSVD: Efficient Low-rank Approximation for Unified Query-Key-Value Weight Compression in Low-Precision Vision-Language Modelscited by 15
  4. 2025IAG: Input-aware Backdoor Attack on VLM-based Visual Groundingcited by 11
  5. 2025SalQ-VLM: Fine-Grained Saliency-Guided Quantization for Vision-Language Modelscited by 7
  6. 2026Fine-Grained Post-Training Quantization for Large Vision Language Models with Quantization-Aware Integrated Gradientscited by 7
  7. 2026HeiSD: Hybrid Speculative Decoding for Embodied Vision-Language-Action Models with Kinematic Awarenesscited by 6
  8. 2026Gated Relational Alignment via Confidence-based Distillation for Efficient VLMscited by 5
  9. 2026MASQuant: Modality-Aware Smoothing Quantization for Multimodal Large Language Modelscited by 5
  10. 2026BATQuant: Outlier-resilient MXFP4 Quantization via Learnable Block-wise Optimizationcited by 5
  11. 2026VEQ: Modality-Adaptive Quantization for MoE Vision-Language Modelscited by 4
  12. 2026Quant Experts: Token-aware Adaptive Error Reconstruction with Mixture of Experts for Large Vision-Language Models Quantizationcited by 3
33 more
  1. 2026SlowBA: An efficiency backdoor attack towards VLM-based GUI agentscited by 3
  2. 2026Rethinking Practical and Efficient Quantization Calibration for Vision-Language Modelscited by 2
  3. 2026Accelerating Multimodal Large Language Models with Prior-Corrected Token Reductioncited by 2
  4. 2025Scene-Aware Vectorized Memory Multi-Agent Framework with Cross-Modal Differentiated Quantization VLMs for Visually Impaired Assistancecited by 1
  5. 2025MC#: Mixture Compressor for Mixture-of-Experts Large Modelscited by 1
  6. 2025Towards Efficient VLMs: Information-Theoretic Driven Compression via Adaptive Structural Pruningcited by 1
  7. 2026HeRo-Q: A General Framework for Stable Low Bit Quantization via Hessian Conditioningcited by 1
  8. 2026QAPruner: Quantization-Aware Vision Token Pruning for Multimodal Large Language Modelscited by 1
  9. 2026SAB-LVLM: Significance-Aware Binarization for Large Vision-Language Modelscited by 1
  10. 2026Rethinking Small VLM Quantization: From Component-Wise Analysis to Hardware-Aware Edge Deploymentcited by 1
  11. 2026BEFORE THE FLIP: Measuring Hidden Score Shifts In Quantized Vision Language Models Before The Answer Changes for Visual Question Answeringcited by 1
  12. 2025Beyond Real Weights: Hypercomplex Representations for Stable Quantizationcited by 0
  13. 2026LinMU: Multimodal Understanding Made Linearcited by 0
  14. 2026MAPLE: Modality-Aware Post-training and Learning Ecosystemcited by 0
  15. 2026Evaluating the Impact of Post-Training Quantization on Reliable VQA with Multimodal LLMscited by 0
  16. 2026DharmaOCR: Specialized Small Language Models for Structured OCR that outperform Open-Source and Commercial Baselinescited by 0
  17. 2026The Cost of Language: Centroid Erasure Exposes and Exploits Modal Competition in Multimodal Language Modelscited by 0
  18. 2026LLaVA-CKD: Bottom-Up Cascaded Knowledge Distillation for Vision-Language Modelscited by 0
  19. 2026MGVQ: Synergizing Multi-dimensional Sensitivity-Aware and Gradient-Hessian Fusion for Vector Quantizationcited by 0
  20. 2026LASER: Loss-Aware Singular-value Decomposition and Rank Allocation for Efficient Low-Precision Vision-Language Modelscited by 0
  21. 2026MorphoQuant: Modality-Aware Quantization for Omni-modal Large Language Modelscited by 0
  22. 2026RotateAttention: RoPE-Aware Rotation and Range Rectification for INT4 Quantized Attention in Video Generationcited by 0
  23. 2026Llama-Mobile: Efficient 2.7-Bit Quantization of VLMscited by 0
  24. 2026Activation Outliers Matter: Robust Recovery for Quantized Multimodal LLMscited by 0
  25. 2026Contribution-Aware Bandwidth Allocation for Multimodal Split Learningcited by 0
  26. 2026MiX: Micro-Inverted-Scaling for End-to-End Low-Bit Vision-Language Model Accelerationcited by 0
  27. 2026Reassessing Global Gradient-Norm Imbalance in BLIP Fine-Tuning Across Physical Domainscited by 0
  28. 2026RGSQ: Riemannian Geometry-Sensitive Quantization for Large Vision-Language Modelscited by 0
  29. 2026GHOST-Q: Towards Studying Grounding Hallucinations Overlooked Under Same-score TradeOffs in Quantized VLMScited by 0
  30. 2026Beyond Reconstruction Loss in Post-Training Quantization: Balanced Fitting for Large Vision-Language Modelscited by 0
  31. 2026When Text Matters: Design Principles for Visual Token Pruning in Vision-Language Modelcited by 0
  32. 2026P4Q: Co-designing Token Pruning and Quantization for Vision-Language Model Accelerationcited by 0
  33. 2026Calibrate the Decisions That Change the Future: On-Policy Post-Training Quantization for Multimodal Large Language Modelscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Bi-VLM: Pushing Ultra-Low Precision Post-Training Quantization Boundaries in Vision-Language Models
  2. 2024Q-VLM: Post-training Quantization for Large Vision-Language Models
  3. 2024P4Q: Learning to Prompt for Quantization in Visual-language Models
  4. 2026Breaking Modality Heterogeneity in Low-Bit Quantization for Large Vision-Language Models
  5. 2026Towards Understanding Best Practices for Quantization of Vision-Language Models
  6. 2025LUQ: Layerwise Ultra-Low Bit Quantization for Multimodal Large Language Models
  7. 2026Saliency-Aware Regularized Quantization Calibration for Large Language Models
  8. 2024LM4LV: A Frozen Large Language Model for Low-level Vision Tasks