mlsanityData to 14 Oct 2026

ShortV: Efficient Multimodal Large Language Models by Freezing Visual Tokens in Ineffective Layers

arXiv 2504.00502 ↗First posted 1 Apr 2025cs.CV cs.CLcited by 14

On the map: Multimodal LLMs / vision-language models → Visual token pruning and efficiency

Abstract (arXiv)

Multimodal Large Language Models (MLLMs) suffer from high computational costs due to their massive size and the large number of visual tokens. In this paper, we investigate layer-wise redundancy in MLLMs by introducing a novel metric, Layer Contribution (LC), which quantifies the impact of a layer's transformations on visual and text tokens, respectively. The calculation of LC involves measuring the divergence in model output that results from removing the layer's transformations on the specified tokens. Our pilot experiment reveals that many layers of MLLMs exhibit minimal contribution during the processing of visual tokens. Motivated by this observation, we propose ShortV, a training-free method that leverages LC to identify ineffective layers, and freezes visual token updates in these layers. Experiments show that ShortV can freeze visual token in approximately 60% of the MLLM layers, thereby dramatically reducing computational costs related to updating visual tokens. For example, it achieves a 50% reduction in FLOPs on LLaVA-NeXT-13B while maintaining superior performance. The code will be publicly available at https://github.com/icip-cas/ShortV

Key facts. Cited by 14 papers in the map (data through 14 Oct 2026). Placed in Visual token pruning and efficiency, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 27 of its 57 references that are in the map. First posted on arXiv 1 Apr 2025.

Builds on

27 of 57 references.

Closest ancestors · 4

Same direction, newest first

  1. 2024TokenPacker: Efficient Visual Projector for Multimodal LLM2 Jul
  2. 2024VoCo-LLaMA: Towards Vision Compression with Large Language Models18 Jun
  3. 2024Boosting Multimodal Large Language Models with Visual Tokens Withdrawal for Rapid Inference9 May
  4. 2024LLaVA-PruMerge: Adaptive Token Reduction for Efficient Large Multimodal Models22 Mar

Other references · 23

  1. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 30,186
  2. 2023GPT-4 Technical Reportcited by 18,396
  3. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 13,028
  4. 2023MiniGPT-4: Enhancing Vision-Language Understanding with Advanced Large Language Modelscited by 2,711
  5. 2023SEED-Bench: Benchmarking Multimodal LLMs with Generative Comprehensioncited by 940
  6. 2024Are We on the Right Way for Evaluating Large Vision-Language Models?cited by 914
  7. 2024DeepSeek-VL: Towards Real-World Vision-Language Understandingcited by 733
  8. 2023Baichuan 2: Open Large-scale Language Modelscited by 670
  9. 2023MiniGPT-v2: large language model as a unified interface for vision-language multi-task learningcited by 625
  10. 2023OpenFlamingo: An Open-Source Framework for Training Large Autoregressive Vision-Language Modelscited by 565
  11. 2023SPHINX: The Joint Mixing of Weights, Tasks, and Visual Embeddings for Multi-modal Large Language Modelscited by 286
  12. 2024AuroraCap: Efficient, Performant Video Detailed Captioning and a New Benchmarkcited by 140
11 more
  1. 2024LaCo: Large Language Model Pruning via Layer Collapsecited by 126
  2. 2024NVLM: Open Frontier-Class Multimodal LLMscited by 125
  3. 2024SLEB: Streamlining LLMs through Redundancy Verification and Elimination of Transformer Blockscited by 97
  4. 2024What Matters in Transformers? Not All Attention is Neededcited by 84
  5. 2024Shortened LLaMA: Depth Pruning for Large Language Models with Comparison of Retraining Methodscited by 82
  6. 2024MovieChat+: Question-aware Sparse Memory for Long Video Question Answeringcited by 73
  7. 2025Vamba: Understanding Hour-Long Videos with Hybrid Mamba-Transformerscited by 36
  8. 2024Streamlining Redundant Layers to Compress Large Language Modelscited by 35
  9. 2024EVLM: An Efficient Vision-Language Model for Visual Understandingcited by 25
  10. 2024Deciphering Cross-Modal Alignment in Large Vision-Language Models with Modality Integration Ratecited by 20
  11. 2025SAISA: Towards Multimodal Large Language Models with Both Training and Inference Efficiencycited by 3

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 4; 2026: 10.

14 papers.

Sort:

By area:

  1. 2026Vision-OPD: Learning to See Fine Details for Multimodal LLMs via On-Policy Self-Distillationcited by 77
  2. 2025Video-MMLU: A Massive Multi-Discipline Lecture Understanding Benchmarkcited by 37
  3. 2025MemSearcher: Training LLMs to Reason, Search and Manage Memory via End-to-End Reinforcement Learningcited by 27
  4. 2025Vision Function Layer in Multimodal LLMscited by 17
  5. 2026ViCA: Efficient Multimodal LLMs with Vision-Only Cross-Attentioncited by 10
  6. 2026ShortOPD: Recovering Pruned LLMs with Short-to-Long On-Policy Distillationcited by 8
  7. 2025TransPrune: Token Transition Pruning for Efficient Large Vision-Language Modelcited by 3
  8. 2026Visual Access Boundaries in Vision-Language Model Reasoningcited by 2
  9. 2026Do Vision Language Models Need to Process Image Tokens?cited by 1
  10. 2026Attend, Transform, or Silence: Operator-Level Visual Skipping for Efficient Multimodal LLM Inferencecited by 1
  11. 2026From Inheritance to Saturation: Disentangling the Evolution of Visual Redundancy for Architecture-Aware MLLM Inference Accelerationcited by 0
  12. 2026Just MLPs: Efficient Visual State Reconstruction for Multimodal Language Modelscited by 0
2 more
  1. 2026SPIDER: Multi-Layer Semantic Token Pruning and Adaptive Sub-Layer Skipping in Multimodal Large Language Modelscited by 0
  2. 2026GeoGR: Spatial-Semantic Token Compression and Group Routing for Efficient 3D VLMscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024Accelerating Multimodal Large Language Models via Dynamic Visual-Token Exit and the Empirical Findings
  2. 2024Treat Visual Tokens as Text? But Your MLLM Only Needs Fewer Efforts to See
  3. 2026One Layer's Trash is Another Layer's Treasure: Adaptive Layer-wise Visual Token Selection in LVLMs
  4. 2024Sparsity Meets Similarity: Leveraging Long-Tail Distribution for Dynamic Optimized Token Representation in Multimodal Large Language Models
  5. 2024ST3: Accelerating Multimodal Large Language Model by Spatial-Temporal Visual Token Trimming
  6. 2025A Comprehensive Study on Visual Token Redundancy for Discrete Diffusion-based Multimodal Large Language Models
  7. 2025LFTR: Learning-Free Token Reduction for Multimodal Large Language Models
  8. 2026CLASP: Class-Adaptive Layer Fusion and Dual-Stage Pruning for Multimodal Large Language Models