mlsanityData to 14 Oct 2026

CASP: Compression of Large Multimodal Models Based on Attention Sparsity

arXiv 2503.05936 ↗First posted 7 Mar 2025cs.CVcited by 5

On the map: Multimodal LLMs / vision-language models → Visual token pruning and efficiency

Abstract (arXiv)

In this work, we propose an extreme compression technique for Large Multimodal Models (LMMs). While previous studies have explored quantization as an efficient post-training compression method for Large Language Models (LLMs), low-bit compression for multimodal models remains under-explored. The redundant nature of inputs in multimodal models results in a highly sparse attention matrix. We theoretically and experimentally demonstrate that the attention matrix's sparsity bounds the compression error of the Query and Key weight matrices. Based on this, we introduce CASP, a model compression technique for LMMs. Our approach performs a data-aware low-rank decomposition on the Query and Key weight matrix, followed by quantization across all layers based on an optimal bit allocation process. CASP is compatible with any quantization technique and enhances state-of-the-art 2-bit quantization methods (AQLM and QuIP#) by an average of 21% on image- and video-language benchmarks.

Key facts. Cited by 5 papers in the map (data through 14 Oct 2026). Placed in Visual token pruning and efficiency, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 39 of its 55 references that are in the map. First posted on arXiv 7 Mar 2025.

Builds on

39 of 55 references.

Closest ancestors · 1

Same direction, newest first

  1. 2023Large Multimodal Model Compression via Efficient Pruning and Distillation at AntGroup10 Dec

Other references · 38

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  2. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,599
  3. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 13,028
  4. 2023Visual Instruction Tuningcited by 7,869
  5. 2023Improved Baselines with Visual Instruction Tuningcited by 4,418
  6. 2016Pointer Sentinel Mixture Modelscited by 3,096
  7. 2024LLaVA-OneVision: Easy Visual Task Transfercited by 2,719
  8. 2023MMBench: Is Your Multi-modal Model an All-around Player?cited by 2,141
  9. 2023MMMU: A Massive Multi-discipline Multimodal Understanding and Reasoning Benchmark for Expert AGIcited by 2,080
  10. 2022Learn to Explain: Multimodal Reasoning via Thought Chains for Science Question Answeringcited by 1,919
  11. 2022GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformerscited by 1,919
  12. 2023MME: A Comprehensive Evaluation Benchmark for Multimodal Large Language Modelscited by 1,652
26 more
  1. 2024Video-MME: The First-Ever Comprehensive Evaluation Benchmark of Multi-modal LLMs in Video Analysiscited by 1,413
  2. 2023Video-ChatGPT: Towards Detailed Video Understanding via Large Vision and Language Modelscited by 1,062
  3. 2023SEED-Bench: Benchmarking Multimodal LLMs with Generative Comprehensioncited by 940
  4. 2022LLM.int8(): 8-bit Matrix Multiplication for Transformers at Scalecited by 906
  5. 2021A Survey of Quantization Methods for Efficient Neural Network Inferencecited by 815
  6. 2021NExT-QA:Next Phase of Question-Answering to Explaining Temporal Actionscited by 744
  7. 2023Distilling Step-by-Step! Outperforming Larger Language Models with Less Training Data and Smaller Model Sizescited by 737
  8. 2023LLM-Pruner: On the Structural Pruning of Large Language Modelscited by 650
  9. 2024KIVI: A Tuning-Free Asymmetric 2bit Quantization for KV Cachecited by 637
  10. 2020Up or Down? Adaptive Rounding for Post-Training Quantizationcited by 621
  11. 2019ActivityNet-QA: A Dataset for Understanding Complex Web Videos via Question Answeringcited by 561
  12. 2022ZeroQuant: Efficient and Affordable Post-Training Quantization for Large-Scale Transformerscited by 526
  13. 2018nocaps: novel object captioning at scalecited by 444
  14. 2023QuIP: 2-Bit Quantization of Large Language Models With Guaranteescited by 414
  15. 2024ShortGPT: Layers in Large Language Models are More Redundant Than You Expectcited by 383
  16. 2024QuIP#: Even Better LLM Quantization with Hadamard Incoherence and Lattice Codebookscited by 378
  17. 2024LMMs-Eval: Reality Check on the Evaluation of Large Multimodal Modelscited by 353
  18. 2024SVD-LLM: Truncation-aware Singular Value Decomposition for Large Language Model Compressioncited by 243
  19. 2024Extreme Compression of Large Language Models via Additive Quantizationcited by 232
  20. 2022Language model compression with weighted low-rank factorizationcited by 195
  21. 2023ASVD: Activation-aware Singular Value Decomposition for Compressing Large Language Modelscited by 169
  22. 2022Overcoming Oscillations in Quantization-Aware Trainingcited by 119
  23. 2023LQ-LoRA: Low-rank Plus Quantized Matrix Decomposition for Efficient Language Model Finetuningcited by 92
  24. 2023Training Transformers with 4-bit Integerscited by 60
  25. 2024MoDeGPT: Modular Decomposition for Large Language Model Compressioncited by 48
  26. 2024GOLD: Generalized Knowledge Distillation via Out-of-Distribution-Guided Language Data Generationcited by 6

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 2; 2026: 3.

5 papers.

  1. 2025Spatial Reasoning with Vision-Language Models in Ego-Centric Multi-View Scenescited by 63
  2. 2026LLaVA-FA: Learning Fourier Approximation for Compressing Large Multimodal Modelscited by 11
  3. 2025From Segments to Scenes: Temporal Understanding for Agentic Autonomous Driving via Vision-Language Modelscited by 2
  4. 2026Cut-ViT: Task-Specific Model Pruning via Gram Anchoring Subspace Consistencycited by 1
  5. 2026Seeing Beyond Redundancy: Task Complexity's Role in Vision Token Specialization in VLLMscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025A Survey of Token Compression for Efficient Multimodal Large Language Models
  2. 2025Enhancing Large Multimodal Models with Adaptive Sparsity and KV Cache Compression
  3. 2026Vision Token Reduction via Attention-Driven Self-Compression for Efficient Multimodal Large Language Models
  4. 2025LUQ: Layerwise Ultra-Low Bit Quantization for Multimodal Large Language Models
  5. 2025Can Visual Input Be Compressed? A Visual Token Compression Benchmark for Large Multimodal Models
  6. 2025LVLM-Compress-Bench: Benchmarking the Broader Impact of Large Vision-Language Model Compression
  7. 2026HybridKV: Hybrid KV Cache Compression for Efficient Multimodal Large Language Model Inference
  8. 2025Lag-Relative Sparse Attention In Long Context Training