mlsanityData to 14 Oct 2026

QSVD: Efficient Low-rank Approximation for Unified Query-Key-Value Weight Compression in Low-Precision Vision-Language Models

arXiv 2510.16292 ↗First posted 18 Oct 2025cs.LGcited by 16

On the map: Multimodal LLMs / vision-language models → Visual token pruning and efficiency

Abstract (arXiv)

Vision-Language Models (VLMs) are integral to tasks such as image captioning and visual question answering, but their high computational cost, driven by large memory footprints and processing time, limits their scalability and real-time applicability. In this work, we propose leveraging Singular-Value Decomposition (SVD) over the joint query (Q), key (K), and value (V) weight matrices to reduce KV cache size and computational overhead. We in addition introduce an efficient rank allocation strategy that dynamically adjusts the SVD rank based on its impact on VLM accuracy, achieving a significant reduction in both memory usage and computational cost. Finally, we extend this approach by applying quantization to both VLM weights and activations, resulting in a highly efficient VLM. Our method outperforms previous approaches that rely solely on quantization or SVD by achieving more than 10% accuracy improvement while consuming less hardware cost, making it better for real-time deployment on resource-constrained devices. We open source our code at hrefhttps://github.com/SAI-Lab-NYU/QSVDtexttthttps://github.com/SAI-Lab-NYU/QSVD.

Key facts. Cited by 16 papers in the map (data through 14 Oct 2026). Placed in Visual token pruning and efficiency, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 53 of its 65 references that are in the map. First posted on arXiv 18 Oct 2025.

Builds on

53 of 65 references.

Closest ancestors · 4

Same direction, newest first

  1. 2025SmolVLM: Redefining small and efficient multimodal models7 Apr
  2. 2024MBQ: Modality-Balanced Quantization for Large Vision-Language Models27 Dec
  3. 2024Q-VLM: Post-training Quantization for Large Vision-Language Models10 Oct
  4. 2024Advancing Multimodal Large Language Models with Quantization-Aware Scale Learning for Efficient Adaptation7 Aug

Other references · 49

  1. 2024The Llama 3 Herd of Modelscited by 14,715
  2. 2023Visual Instruction Tuningcited by 7,869
  3. 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,137
  4. 2022BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generationcited by 4,322
  5. 2024Qwen2-VL: Enhancing Vision-Language Model's Perception of the World at Any Resolutioncited by 4,234
  6. 2022Learn to Explain: Multimodal Reasoning via Thought Chains for Science Question Answeringcited by 1,919
  7. 2022GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformerscited by 1,919
  8. 2022SmoothQuant: Accurate and Efficient Post-Training Quantization for Large Language Modelscited by 1,386
  9. 2023GQA: Training Generalized Multi-Query Transformer Models from Multi-Head Checkpointscited by 1,346
  10. 2023AWQ: Activation-aware Weight Quantization for LLM Compression and Accelerationcited by 1,298
  11. 2018VizWiz Grand Challenge: Answering Visual Questions from Blind Peoplecited by 858
  12. 2023Vision-Language Models for Vision Tasks: A Surveycited by 839
37 more
  1. 2019Fast Transformer Decoding: One Write-Head is All You Needcited by 717
  2. 2024PaliGemma: A versatile 3B VLM for transfercited by 696
  3. 2024QuaRot: Outlier-Free 4-Bit Inference in Rotated LLMscited by 634
  4. 2023HallusionBench: An Advanced Diagnostic Suite for Entangled Language Hallucination and Visual Illusion in Large Vision-Language Modelscited by 583
  5. 2019Unified Vision-Language Pre-Training for Image Captioning and VQAcited by 494
  6. 2024SpinQuant: LLM quantization with learned rotationscited by 471
  7. 2024VLMEvalKit: An Open-Source Toolkit for Evaluating Large Multi-Modality Modelscited by 455
  8. 2023OmniQuant: Omnidirectionally Calibrated Quantization for Large Language Modelscited by 424
  9. 2024QuIP#: Even Better LLM Quantization with Hadamard Incoherence and Lattice Codebookscited by 378
  10. 2024A Survey on Hallucination in Large Vision-Language Modelscited by 357
  11. 2023SqueezeLLM: Dense-and-Sparse Quantizationcited by 294
  12. 2024SVD-LLM: Truncation-aware Singular Value Decomposition for Large Language Model Compressioncited by 243
  13. 2021Scaling Up Vision-Language Pre-training for Image Captioningcited by 196
  14. 2022Language model compression with weighted low-rank factorizationcited by 195
  15. 2021VisualGPT: Data-efficient Adaptation of Pretrained Language Models for Image Captioningcited by 191
  16. 2024SVDQuant: Absorbing Outliers by Low-Rank Components for 4-Bit Diffusion Modelscited by 170
  17. 2023ASVD: Activation-aware Singular Value Decomposition for Compressing Large Language Modelscited by 169
  18. 2024DuQuant: Distributing Outliers via Dual Transformation Makes Stronger Quantized LLMscited by 167
  19. 2024TinyLLaVA: A Framework of Small-scale Large Multimodal Modelscited by 164
  20. 2024Divide, Conquer and Combine: A Training-Free Framework for High-Resolution Image Perception in Multimodal Large Language Modelscited by 161
  21. 2023TinyGPT-V: Efficient Multimodal Large Language Model via Small Backbonescited by 77
  22. 2024AffineQuant: Affine Transformation Quantization for Large Language Modelscited by 67
  23. 2025SVD-LLM V2: Optimizing Singular Value Truncation for Large Language Model Compressioncited by 67
  24. 2024Palu: Compressing KV-Cache with Low-Rank Projectioncited by 64
  25. 2023Qilin-Med-VL: Towards Chinese Large Vision-Language Model for General Healthcarecited by 62
  26. 2023Training Transformers with 4-bit Integerscited by 60
  27. 2025Dobi-SVD: Differentiable SVD for LLM Compression and Some New Perspectivescited by 48
  28. 2024Surgical-LVLM: Learning to Adapt Large Vision-Language Model for Grounded Visual Question Answering in Robotic Surgerycited by 39
  29. 2024Effectively Compress KV Heads for LLMcited by 30
  30. 2025JARVIS-VLA: Post-Training Large-Scale Vision Language Models to Play Visual Games with Keyboards and Mousecited by 28
  31. 2024SearchLVLMs: A Plug-and-Play Framework for Augmenting Large Vision-Language Models by Searching Up-to-Date Internet Knowledgecited by 20
  32. 2024VLRM: Vision-Language Models act as Reward Models for Image Captioningcited by 18
  33. 2024DFRot: Achieving Outlier-Free and Massive Activation-Free for Rotated LLMs with Refined Rotationcited by 17
  34. 2025AdaSVD: Adaptive Singular Value Decomposition for Large Language Modelscited by 15
  35. 2024Leveraging Large Vision-Language Model as User Intent-aware Encoder for Composed Image Retrievalcited by 13
  36. 2024VideoGameBunny: Towards vision assistants for video gamescited by 11
  37. 2024Rotated Runtime Smooth: Training-Free Activation Smoother for accurate INT4 inferencecited by 5

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 16.

16 papers.

Sort:

By area:

  1. 2026WSVD: Weighted Low-Rank Approximation for Fast and Efficient Execution of Low-Precision Vision-Language Modelscited by 3
  2. 2026FlashSVD v1.5: Making Low-Rank Transformers Inference Actually Fastcited by 3
  3. 2026Understanding Calibration and Truncation Error Propagation in Training-Free Low-Rank Compression for LLMscited by 2
  4. 2026Concatenated Matrix SVD: Compression Bounds, Incremental Approximation, and Error-Constrained Clusteringcited by 1
  5. 2026Breaking Modality Heterogeneity in Low-Bit Quantization for Large Vision-Language Modelscited by 1
  6. 2026GPTQ-intrinsic LoRA: A Near-optimal Algorithm for Low-precision Quantization with Low-rank Adaptationcited by 1
  7. 2026CodeQuant: Unified Clustering and Quantization for Enhanced Outlier Smoothing in Low-Precision Mixture-of-Expertscited by 0
  8. 2026IO-SVD: Input-Output Whitened SVD for Adaptive-Rank LLM Compressioncited by 0
  9. 2026LASER: Loss-Aware Singular-value Decomposition and Rank Allocation for Efficient Low-Precision Vision-Language Modelscited by 0
  10. 2026BitNet Text Embeddingscited by 0
  11. 2026Quasi-SVD: Learning a Lie-constrained matrix factorisation for real-time imagingcited by 0
  12. 2026Federated Foundation Models Fine-Tuning with Heterogeneous Compressed Clientscited by 0
4 more
  1. 2026LowRankArena: A Standardized Evaluation Platform for SVD-Based LLM Compressioncited by 0
  2. 2026Learning Functional Subspaces for Neural Network Compressioncited by 0
  3. 2026Sequential Functional Structured Tucker Compression for Large Language Model Attentionscited by 0
  4. 2026Differentiable Bit-Widths: Co-optimizing Pruning and Quantization via SVD for Ultra-Efficient LLM Compressioncited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Bi-VLM: Pushing Ultra-Low Precision Post-Training Quantization Boundaries in Vision-Language Models
  2. 2024VL-Cache: Sparsity and Modality-Aware KV Cache Compression for Vision-Language Model Inference Acceleration
  3. 2025A Survey on Efficient Vision-Language Models
  4. 2025Benchmarking and Enhancing VLM for Compressed Image Understanding
  5. 2024ZipVL: Efficient Large Vision-Language Models with Dynamic Token Sparsification
  6. 2026Llama-Mobile: Efficient 2.7-Bit Quantization of VLMs
  7. 2026Towards Understanding Best Practices for Quantization of Vision-Language Models
  8. 2026KVCapsule: Efficient Sequential KV Cache Compression for Vision-Language Models with Asymmetric Redundancy