# Short-LVLM: Compressing and Accelerating Large Vision-Language Models by Pruning Redundant Layers

arXiv: https://arxiv.org/abs/2507.23362 · HTML: https://mlsanity.com/paper/2507.23362/

## Key facts

- Cited by 7 papers in the map (data through 14 Oct 2026).
- Placed in Visual token pruning and efficiency, one of 16 directions in Multimodal LLMs / vision-language models.
- Builds on 25 of its 41 references that are in the map.
- First posted on arXiv 31 Jul 2025.

## Placed in

- Multimodal LLMs / vision-language models / [Visual token pruning and efficiency](https://mlsanity.com/mllm/visual-token-pruning-and-efficiency/)

## Builds on (most cited first, 25 of 25)

- [LLaMA: Open and Efficient Foundation Language Models](https://mlsanity.com/paper/2302.13971/) (2023)
- [Scaling Laws for Neural Language Models](https://mlsanity.com/paper/2001.08361/) (2020)
- [Qwen2.5 Technical Report](https://mlsanity.com/paper/2412.15115/) (2024)
- [Qwen Technical Report](https://mlsanity.com/paper/2309.16609/) (2023)
- [MMBench: Is Your Multi-modal Model an All-around Player?](https://mlsanity.com/paper/2307.06281/) (2023)
- [Learning both Weights and Connections for Efficient Neural Networks](https://mlsanity.com/paper/1506.02626/) (2015)
- [Learn to Explain: Multimodal Reasoning via Thought Chains for Science Question Answering](https://mlsanity.com/paper/2209.09513/) (2022)
- [GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformers](https://mlsanity.com/paper/2210.17323/) (2022)
- [A-OKVQA: A Benchmark for Visual Question Answering using World Knowledge](https://mlsanity.com/paper/2206.01718/) (2022)
- [mPLUG-Owl2: Revolutionizing Multi-modal Large Language Model with Modality Collaboration](https://mlsanity.com/paper/2311.04257/) (2023)
- [The Unreasonable Ineffectiveness of the Deeper Layers](https://mlsanity.com/paper/2403.17887/) (2024)
- [Boosting Multimodal Large Language Models with Visual Tokens Withdrawal for Rapid Inference](https://mlsanity.com/paper/2405.05803/) (2024)
- [SLEB: Streamlining LLMs through Redundancy Verification and Elimination of Transformer Blocks](https://mlsanity.com/paper/2402.09025/) (2024)
- [UPop: Unified and Progressive Pruning for Compressing Vision-Language Transformers](https://mlsanity.com/paper/2301.13741/) (2023)
- [Nullu: Mitigating Object Hallucinations in Large Vision-Language Models via HalluSpace Projection](https://mlsanity.com/paper/2412.13817/) (2024)
- [Playing Lottery Tickets with Vision and Language](https://mlsanity.com/paper/2104.11832/) (2021)
- [MoPE-CLIP: Structured Pruning for Efficient Vision-Language Models with Module-wise Pruning Error Metric](https://mlsanity.com/paper/2403.07839/) (2024)
- [Sparsity May Cry: Let Us Fail (Current) Sparse Neural Networks Together!](https://mlsanity.com/paper/2303.02141/) (2023)
- [Why Random Pruning Is All We Need to Start Sparse](https://mlsanity.com/paper/2210.02412/) (2022)
- [Gradient-Free Structured Pruning with Unlabeled Data](https://mlsanity.com/paper/2303.04185/) (2023)
- [Octopus: Alleviating Hallucination via Dynamic Contrastive Decoding](https://mlsanity.com/paper/2503.00361/) (2025)
- [Routing Experts: Learning to Route Dynamic Experts in Multi-modal Large Language Models](https://mlsanity.com/paper/2407.14093/) (2024)
- [Residual Alignment: Uncovering the Mechanisms of Residual Networks](https://mlsanity.com/paper/2401.09018/) (2024)
- [S3C: Semi-Supervised VQA Natural Language Explanation via Self-Critical Learning](https://mlsanity.com/paper/2309.02155/) (2023)
- [Transformer Block Coupling and its Correlation with Generalization in LLMs](https://mlsanity.com/paper/2407.07810/) (2024)

## Built on by (most cited first, 7 of 7)

- [VisionSelector: End-to-End Learnable Visual Token Compression for Efficient Multimodal LLMs](https://mlsanity.com/paper/2510.16598/) (2025)
- [Seeing the End at Step Zero: Accelerating Diffusion MLLMs via MLP Sparsity-Aware Truncation](https://mlsanity.com/paper/2607.14557/) (2026)
- [HiSC: Hierarchical Spatial Clustering Token Compression for Efficient 3D Scene Understanding](https://mlsanity.com/paper/2608.04610/) (2026)
- [INTERLACE: Interleaved Layer Pruning and Efficient Adaptation in Large Vision-Language Models](https://mlsanity.com/paper/2511.19676/) (2025)
- [LinMU: Multimodal Understanding Made Linear](https://mlsanity.com/paper/2601.01322/) (2026)
- [HiPP-Prune: Hierarchical Preference-Conditioned Structured Pruning for Vision-Language Models](https://mlsanity.com/paper/2603.06270/) (2026)
- [Searching for Task-Specific Vision Paths: Evolutionary Block Pruning Across Vision-Language Models](https://mlsanity.com/paper/2607.17052/) (2026)

## Abstract (arXiv)

Although large vision-language models (LVLMs) have demonstrated impressive capabilities in multi-modal understanding and reasoning, their practical applications are still limited by massive model parameters and high computational costs. Recent efforts from natural language processing (NLP) have shown the effectiveness of layer pruning, offering a plausible training-free compression solution. However, due to the modality divergence between vision and language, it is unclear whether these NLP techniques are still effective in LVLMs. In this paper, we empirically prove that directly applying these layer pruning methods to LVLMs is ineffective. Through extensive experiments, we find that non-essential vision-language (VL) tokens and inter-layer feature gaps pose critical challenges to pruning layers in LVLMs. Based on these insights, we propose a novel framework Short-LVLM (SVL) that can utilize important VL tokens and mitigate the layer-wise feature gaps. Notably, Short-LVLM not only achieves a superior trade-off between performance and efficiency but also exhibits several potential advantages, i.e., training-free, model-agnostic, and highly compatible. The code for this work is publicly available at https://github.com/ASGO-MM/Short-LVLM.