mlsanityData to 7 Oct 2026

Rethinking Pruning for Vision-Language Models: Strategies for Effective Sparsity and Performance Restoration

arXiv 2404.02424 ↗First posted 3 Apr 2024cs.LG cs.CVcited by 4

On the map: LLM reasoning → Multimodal reasoning

also in 1 more
Abstract (arXiv)

Vision-Language Models (VLMs) integrate information from multiple modalities and have shown remarkable success across various tasks. However, deploying large-scale VLMs in resource-constrained scenarios is challenging. Pruning followed by finetuning offers a potential solution but remains underexplored for VLMs. This study addresses two key questions: how to distribute sparsity across different modality-specific models, and how to restore the performance of pruned sparse VLMs. Our preliminary studies identified two effective pruning settings: applying the same sparsity to both vision and language models, and pruning only the language models. While LoRA finetuning aims to restore sparse models, it faces challenges due to incompatibility with sparse models, disrupting the pruned sparsity. To overcome these issues, we propose SparseLoRA, which applies sparsity directly to LoRA weights. Our experimental results demonstrate significant improvements, including an 11.3% boost under 2:4 sparsity and a 47.6% enhancement under unstructured 70% sparsity. Code is released at: urlhttps://github.com/Shwai-He/VLM-Compression.

Key facts. Cited by 4 papers in the map (data through 7 Oct 2026). Placed in Multimodal reasoning, one of 17 directions in LLM reasoning. Builds on 30 of its 58 references that are in the map. First posted on arXiv 3 Apr 2024.

Builds on

30 of 58 references.

References · 30

  1. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  2. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  3. 2020Language Models are Few-Shot Learnerscited by 32,199
  4. 2023GPT-4 Technical Reportcited by 18,132
  5. 2015Distilling the Knowledge in a Neural Networkcited by 10,076
  6. 2019Parameter-Efficient Transfer Learning for NLPcited by 4,195
  7. 2023InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuningcited by 2,855
  8. 2023MiniGPT-4: Enhancing Vision-Language Understanding with Advanced Large Language Modelscited by 2,689
  9. 2015Deep Compression: Compressing Deep Neural Networks with Pruning, Trained Quantization and Huffman Codingcited by 2,291
  10. 2018The Lottery Ticket Hypothesis: Finding Sparse, Trainable Neural Networkscited by 2,250
  11. 2024Mixtral of Expertscited by 1,785
  12. 2022Make-A-Video: Text-to-Video Generation without Text-Video Datacited by 1,725
18 more
  1. 2023SparseGPT: Massive Language Models Can Be Accurately Pruned in One-Shotcited by 1,090
  2. 2023LIMA: Less Is More for Alignmentcited by 1,080
  3. 2023A Survey on Multimodal Large Language Modelscited by 898
  4. 2023A Simple and Effective Pruning Approach for Large Language Modelscited by 776
  5. 2013An Empirical Investigation of Catastrophic Forgetting in Gradient-Based Neural Networkscited by 711
  6. 2020Pruning neural networks without any data by iteratively conserving synaptic flowcited by 475
  7. 2024Sora: A Review on Background, Technology, Limitations, and Opportunities of Large Vision Modelscited by 467
  8. 2020The Lottery Ticket Hypothesis for Pre-trained BERT Networkscited by 275
  9. 2020AdapterDrop: On the Efficiency of Adapters in Transformerscited by 223
  10. 2021Accelerating Sparse Deep Neural Networkscited by 213
  11. 2021Does Knowledge Distillation Really Work?cited by 187
  12. 2020Soft Threshold Weight Reparameterization for Learnable Sparsitycited by 150
  13. 2021Group Fisher Pruning for Practical Network Compressioncited by 130
  14. 2021LAFITE: Towards Language-Free Training for Text-to-Image Generationcited by 126
  15. 2023Dynamic Sparse No Training: Training-Free Fine-tuning for Sparse LLMscited by 61
  16. 2024Everybody Prune Now: Structured Pruning of LLMs with only Forward Passescited by 42
  17. 2023A Comprehensive Analysis of Adapter Efficiencycited by 8
  18. 2022SD-Conv: Towards the Parameter-Efficiency of Dynamic Convolutioncited by 2

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 3; 2026: 1.

4 papers.

  1. 2025TAMP: Token-Adaptive Layerwise Pruning in Multimodal Large Language Modelscited by 9
  2. 2025Taming Vision-Language Models for Medical Image Analysis: A Comprehensive Reviewcited by 9
  3. 2025Understanding and Harnessing Sparsity in Unified Multimodal Modelscited by 4
  4. 2026HiPP-Prune: Hierarchical Preference-Conditioned Structured Pruning for Vision-Language Modelscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026Prune Once: Retraining-Free Task-Agnostic Pruning for Vision-Language Models
  2. 2023ECoFLaP: Efficient Coarse-to-Fine Layer-Wise Pruning for Vision-Language Models
  3. 2025Short-LVLM: Compressing and Accelerating Large Vision-Language Models by Pruning Redundant Layers
  4. 2026SlimVLM: Sensitivity-aware Dynamic Structured Pruning with Adaptive Visual Token Selection for Efficient Vision-Language Models
  5. 2026Structural Pruning of Large Vision Language Models: A Comprehensive Study on Pruning Dynamics, Recovery, and Data Efficiency
  6. 2022EfficientVLM: Fast and Accurate Vision-Language Models via Knowledge Distillation and Modal-adaptive Pruning
  7. 2026Mostly Text, Smart Visuals: Asymmetric Text-Visual Pruning for Large Vision-Language Models
  8. 2024SparseVLM: Visual Token Sparsification for Efficient Vision-Language Model Inference