# [CLS] Token Tells Everything Needed for Training-free Efficient MLLMs

arXiv: https://arxiv.org/abs/2412.05819 · HTML: https://mlsanity.com/paper/2412.05819/

## Key facts

- Cited by 26 papers in the map (data through 14 Oct 2026).
- Placed in Visual token pruning and efficiency, one of 16 directions in Multimodal LLMs / vision-language models.
- Builds on 39 of its 73 references that are in the map.
- First posted on arXiv 8 Dec 2024.

## Placed in

- Multimodal LLMs / vision-language models / [Visual token pruning and efficiency](https://mlsanity.com/mllm/visual-token-pruning-and-efficiency/)

## Builds on (most cited first, 30 of 39)

- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [The Llama 3 Herd of Models](https://mlsanity.com/paper/2407.21783/) (2024)
- [LLaMA: Open and Efficient Foundation Language Models](https://mlsanity.com/paper/2302.13971/) (2023)
- [Llama 2: Open Foundation and Fine-Tuned Chat Models](https://mlsanity.com/paper/2307.09288/) (2023)
- [BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Models](https://mlsanity.com/paper/2301.12597/) (2023)
- [MobileNets: Efficient Convolutional Neural Networks for Mobile Vision Applications](https://mlsanity.com/paper/1704.04861/) (2017)
- [Qwen Technical Report](https://mlsanity.com/paper/2309.16609/) (2023)
- [InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuning](https://mlsanity.com/paper/2305.06500/) (2023)
- [MiniGPT-4: Enhancing Vision-Language Understanding with Advanced Large Language Models](https://mlsanity.com/paper/2304.10592/) (2023)
- [Qwen-VL: A Versatile Vision-Language Model for Understanding, Localization, Text Reading, and Beyond](https://mlsanity.com/paper/2308.12966/) (2023)
- [MMBench: Is Your Multi-modal Model an All-around Player?](https://mlsanity.com/paper/2307.06281/) (2023)
- [PaLM-E: An Embodied Multimodal Language Model](https://mlsanity.com/paper/2303.03378/) (2023)
- [Evaluating Object Hallucination in Large Vision-Language Models](https://mlsanity.com/paper/2305.10355/) (2023)
- [MM-Vet: Evaluating Large Multimodal Models for Integrated Capabilities](https://mlsanity.com/paper/2308.02490/) (2023)
- [ChatGLM: A Family of Large Language Models from GLM-130B to GLM-4 All Tools](https://mlsanity.com/paper/2406.12793/) (2024)
- [Token Merging: Your ViT But Faster](https://mlsanity.com/paper/2210.09461/) (2022)
- [Gemini: A Family of Highly Capable Multimodal Models](https://mlsanity.com/paper/2312.11805/) (2023)
- [CogVLM: Visual Expert for Pretrained Language Models](https://mlsanity.com/paper/2311.03079/) (2023)
- [An Image is Worth 1/2 Tokens After Layer 2: Plug-and-Play Inference Acceleration for Large Vision-Language Models](https://mlsanity.com/paper/2403.06764/) (2024)
- [YOLOv10: Real-Time End-to-End Object Detection](https://mlsanity.com/paper/2405.14458/) (2024)
- [LLaVA-PruMerge: Adaptive Token Reduction for Efficient Large Multimodal Models](https://mlsanity.com/paper/2403.15388/) (2024)
- [Mini-Gemini: Mining the Potential of Multi-modality Vision Language Models](https://mlsanity.com/paper/2403.18814/) (2024)
- [Not All Patches are What You Need: Expediting Vision Transformers via Token Reorganizations](https://mlsanity.com/paper/2202.07800/) (2022)
- [MoE-LLaVA: Mixture of Experts for Large Vision-Language Models](https://mlsanity.com/paper/2401.15947/) (2024)
- [DriveMLM: Aligning Multi-Modal Large Language Models with Behavioral Planning States for Autonomous Driving](https://mlsanity.com/paper/2312.09245/) (2023)
- [ALLaVA: Harnessing GPT4V-Synthesized Data for Lite Vision-Language Models](https://mlsanity.com/paper/2402.11684/) (2024)
- [MobileVLM V2: Faster and Stronger Baseline for Vision Language Model](https://mlsanity.com/paper/2402.03766/) (2024)
- [MobileVLM : A Fast, Strong and Open Vision Language Assistant for Mobile Devices](https://mlsanity.com/paper/2312.16886/) (2023)
- [Efficient Multimodal Learning from Data-centric Perspective](https://mlsanity.com/paper/2402.11530/) (2024)
- [Large Language Models for Robotics: Opportunities, Challenges, and Perspectives](https://mlsanity.com/paper/2401.04334/) (2024)

## Built on by (most cited first, 26 of 26)

- [HoliTom: Holistic Token Merging for Fast Video Large Language Models](https://mlsanity.com/paper/2505.21334/) (2025)
- [FastVID: Dynamic Density Pruning for Fast Video Large Language Models](https://mlsanity.com/paper/2503.11187/) (2025)
- [Don't Just Chase "Highlighted Tokens" in MLLMs: Revisiting Visual Holistic Context Retention](https://mlsanity.com/paper/2510.02912/) (2025)
- [A Survey of Multimodal Retrieval-Augmented Generation](https://mlsanity.com/paper/2504.08748/) (2025)
- [Shifting AI Efficiency From Model-Centric to Data-Centric Compression](https://mlsanity.com/paper/2505.19147/) (2025)
- [LLaVA-Scissor: Token Compression with Semantic Connected Components for Video LLMs](https://mlsanity.com/paper/2506.21862/) (2025)
- [VFlowOpt: A Token Pruning Framework for LMMs with Visual Information Flow-Guided Optimization](https://mlsanity.com/paper/2508.05211/) (2025)
- [TokenCarve: Information-Preserving Visual Token Compression in Multimodal Large Language Models](https://mlsanity.com/paper/2503.10501/) (2025)
- [VisionTrim: Unified Vision Token Compression for Training-Free MLLM Acceleration](https://mlsanity.com/paper/2601.22674/) (2026)
- [Beyond Intermediate States: Explaining Visual Redundancy through Language](https://mlsanity.com/paper/2503.20540/) (2025)
- [B-VLLM: A Vision Large Language Model with Balanced Spatio-Temporal Tokens](https://mlsanity.com/paper/2412.09919/) (2024)
- [Unified Spatiotemporal Token Compression for Video-LLMs at Ultra-Low Retention](https://mlsanity.com/paper/2603.21957/) (2026)
- [Achieving More with Less: Additive Prompt Tuning for Rehearsal-Free Class-Incremental Learning](https://mlsanity.com/paper/2503.07979/) (2025)
- [Attention Debiasing for Token Pruning in Vision Language Models](https://mlsanity.com/paper/2508.17807/) (2025)
- [Geometry-Guided 3D Visual Token Pruning for Video-Language Models](https://mlsanity.com/paper/2604.18260/) (2026)
- [AdaTP: Attention-Debiased Token Pruning for Video Large Language Models](https://mlsanity.com/paper/2505.20100/) (2025)
- [METEOR: Multi-Encoder Collaborative Token Pruning for Efficient Vision Language Models](https://mlsanity.com/paper/2507.20842/) (2025)
- [PruneHal: Reducing Hallucinations in Multi-modal Large Language Models through Adaptive KV Cache Pruning](https://mlsanity.com/paper/2510.19183/) (2025)
- [CoMI-IRL: Contrastive Multi-Intention Inverse Reinforcement Learning](https://mlsanity.com/paper/2602.07496/) (2026)
- [One Layer's Trash is Another Layer's Treasure: Adaptive Layer-wise Visual Token Selection in LVLMs](https://mlsanity.com/paper/2606.14277/) (2026)
- [Parallel Vision Token Scheduling for Fast and Accurate Multimodal LMMs Inference](https://mlsanity.com/paper/2511.18875/) (2025)
- [ConsensusDrop: Fusing Visual and Cross-Modal Saliency for Efficient Vision Language Models](https://mlsanity.com/paper/2602.00946/) (2026)
- [GATech at AbjadMed: Bidirectional Encoders vs. Causal Decoders: Insights from 82-Class Arabic Medical Classification](https://mlsanity.com/paper/2603.10008/) (2026)
- [Efficient3D: A Unified Framework for Adaptive and Debiased Token Reduction in 3D MLLMs](https://mlsanity.com/paper/2604.02689/) (2026)
- [Sink-Token-Aware Pruning for Fine-Grained Video Understanding in Efficient Video LLMs](https://mlsanity.com/paper/2604.20937/) (2026)
- [SPIDER: Multi-Layer Semantic Token Pruning and Adaptive Sub-Layer Skipping in Multimodal Large Language Models](https://mlsanity.com/paper/2609.34977/) (2026)

## Abstract (arXiv)

Multimodal Large Language Models (MLLMs) have recently demonstrated strong performance across a wide range of vision-language tasks, garnering significant attention in the computer vision. However, their efficient deployment remains a substantial challenge due to high computational costs and memory requirements. Recognizing the redundancy of information within the vision modality, recent studies have explored methods for compressing visual tokens in MLLMs to enhance efficiency in a training-free manner. Despite their effectiveness, existing methods like Fast rely on the attention between visual tokens and prompt text tokens as the importance indicator, overlooking the relevance to response text and thus introducing perception bias. In this paper, we demonstrate that in MLLMs, the [CLS] token in the visual encoder inherently knows which visual tokens are important for MLLMs. Building on this prior, we introduce a simple yet effective method for train-free visual token compression, called VTC-CLS. Firstly, it leverages the attention score of the [CLS] token on visual tokens as an importance indicator for pruning visual tokens. Besides, we also explore ensembling the importance scores derived by the [CLS] token from different layers to capture the key visual information more comprehensively. Extensive experiments demonstrate that our VTC-CLS achieves the state-of-the-art performance across various tasks compared with baseline methods. It also brings notably less computational costs in a training-free manner, highlighting its effectiveness and superiority. Code and models are available at urlhttps://github.com/THU-MIG/VTC-CLS.