mlsanityData to 14 Oct 2026

TokenCarve: Information-Preserving Visual Token Compression in Multimodal Large Language Models

arXiv 2503.10501 ↗First posted 13 Mar 2025cs.CVcited by 19

On the map: Multimodal LLMs / vision-language models → Visual token pruning and efficiency

also in 1 more
Abstract (arXiv)

Multimodal Large Language Models (MLLMs) are becoming increasingly popular, while the high computational cost associated with multimodal data input, particularly from visual tokens, poses a significant challenge. Existing training-based token compression methods improve inference efficiency but require costly retraining, while training-free methods struggle to maintain performance when aggressively reducing token counts. In this study, we reveal that the performance degradation of MLLM closely correlates with the accelerated loss of information in the attention output matrix. This insight introduces a novel information-preserving perspective, making it possible to maintain performance even under extreme token compression. Based on this finding, we propose TokenCarve, a training-free, plug-and-play, two-stage token compression framework. The first stage employs an Information-Preservation-Guided Selection (IPGS) strategy to prune low-information tokens, while the second stage further leverages IPGS to guide token merging, minimizing information loss. Extensive experiments on 11 datasets and 2 model variants demonstrate the effectiveness of TokenCarve. It can even reduce the number of visual tokens to 22.2% of the original count, achieving a 1.23x speedup in inference, a 64% reduction in KV cache storage, and only a 1.54% drop in accuracy. Our code is available at https://github.com/ShawnTan86/TokenCarve.

Key facts. Cited by 19 papers in the map (data through 14 Oct 2026). Placed in Visual token pruning and efficiency, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 37 of its 44 references that are in the map. First posted on arXiv 13 Mar 2025.

Builds on

37 of 44 references.

Closest ancestors · 16

Same direction, newest first

  1. 2025LLaVA-Mini: Efficient Image and Video Large Multimodal Models with One Vision Token7 Jan
  2. 2025FOLDER: Accelerating Multi-modal Large Language Models with Enhanced Performance5 Jan
  3. 2025What Kind of Visual Tokens Do We Need? Training-free Visual Token Pruning for Multi-modal Large Language Models from the Perspective of Graph4 Jan
  4. 2024[CLS] Token Tells Everything Needed for Training-free Efficient MLLMs8 Dec
  5. 2024VisionZip: Longer is Better but Not Necessary in Vision Language Models5 Dec
  6. 2024p-MoD: Building Mixture-of-Depths MLLMs via Progressive Ratio Decay5 Dec
  7. 2024FlashSloth: Lightning Multimodal Large Language Models via Embedded Visual Compression5 Dec
  8. 2024Dynamic-LLaVA: Efficient Multimodal Large Language Models via Dynamic Vision-language Context Sparsification1 Dec
  9. 2024Multi-Stage Vision Token Dropping: Towards Efficient Multimodal Large Language Model16 Nov
  10. 2024Inference Optimal VLMs Need Fewer Visual Tokens and More Parameters5 Nov
  11. 2024PyramidDrop: Accelerating Your Large Vision-Language Models via Pyramid Visual Redundancy Reduction22 Oct
  12. 2024SparseVLM: Visual Token Sparsification for Efficient Vision-Language Model Inference6 Oct
4 more
  1. 2024Efficient Large Multi-modal Models via Visual Context Compression28 Jun
  2. 2024DeepStack: Deeply Stacking Visual Tokens is Surprisingly Simple and Effective for LMMs6 Jun
  3. 2024LLaVA-PruMerge: Adaptive Token Reduction for Efficient Large Multimodal Models22 Mar
  4. 2024An Image is Worth 1/2 Tokens After Layer 2: Plug-and-Play Inference Acceleration for Large Vision-Language Models11 Mar

Other references · 21

  1. 2017Attention Is All You Needcited by 60,361
  2. 2020Language Models are Few-Shot Learnerscited by 32,419
  3. 2023GPT-4 Technical Reportcited by 18,396
  4. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 14,132
  5. 2023Visual Instruction Tuningcited by 7,869
  6. 2025DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learningcited by 6,330
  7. 2024Qwen2-VL: Enhancing Vision-Language Model's Perception of the World at Any Resolutioncited by 4,234
  8. 2022Constitutional AI: Harmlessness from AI Feedbackcited by 3,050
  9. 2023InternVL: Scaling up Vision Foundation Models and Aligning for Generic Visual-Linguistic Taskscited by 2,565
  10. 2016Making the V in VQA Matter: Elevating the Role of Image Understanding in Visual Question Answeringcited by 2,516
  11. 2023MMBench: Is Your Multi-modal Model an All-around Player?cited by 2,141
  12. 2022Learn to Explain: Multimodal Reasoning via Thought Chains for Science Question Answeringcited by 1,919
9 more
  1. 2023Evaluating Object Hallucination in Large Vision-Language Modelscited by 1,729
  2. 2019Towards VQA Models That Can Readcited by 1,718
  3. 2023MME: A Comprehensive Evaluation Benchmark for Multimodal Large Language Modelscited by 1,652
  4. 2020Linformer: Self-Attention with Linear Complexitycited by 1,474
  5. 2023MM-Vet: Evaluating Large Multimodal Models for Integrated Capabilitiescited by 1,239
  6. 2022Token Merging: Your ViT But Fastercited by 1,021
  7. 2022A-OKVQA: A Benchmark for Visual Question Answering using World Knowledgecited by 710
  8. 2024MADTP: Multimodal Alignment-Guided Dynamic Token Pruning for Accelerating Vision-Language Transformercited by 54
  9. 2024Once for Both: Single Stage of Importance and Sparsity Search for Vision Transformer Compressioncited by 8

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 9; 2026: 10.

19 papers.

Sort:

By area:

  1. 2025VScan: Rethinking Visual Token Reduction for Efficient Large Vision-Language Modelscited by 41
  2. 2025A Survey of Token Compression for Efficient Multimodal Large Language Modelscited by 25
  3. 2025Efficient Multi-modal Large Language Models via Progressive Consistency Distillationcited by 22
  4. 2025Why 1 + 1 < 1 in Visual Token Pruning: Beyond Naive Integration via Multi-Objective Balanced Coveringcited by 11
  5. 2026ViCA: Efficient Multimodal LLMs with Vision-Only Cross-Attentioncited by 10
  6. 2025CLIP-MG: Guiding Semantic Attention with Skeletal Pose Features and RGB Data for Micro-Gesture Recognition on the iMiGUE Datasetcited by 4
  7. 2025Attention Debiasing for Token Pruning in Vision Language Modelscited by 4
  8. 2025Revisiting MLLM Token Technology through the Lens of Classical Visual Codingcited by 3
  9. 2026Keep What Audio Cannot Say: Context-Preserving Token Pruning for Omni-LLMscited by 3
  10. 2026OmniRefine: Alignment-Aware Cooperative Compression for Efficient Omnimodal Large Language Modelscited by 3
  11. 2025Task-Related Token Compression in Multimodal Large Language Models from an Explainability Perspectivecited by 2
  12. 2026Compression Tells Intelligence: Visual Coding, Visual Token Technology, and the Unificationcited by 2
7 more
  1. 2025SVAC: Scaling Is All You Need For Referring Video Object Segmentationcited by 1
  2. 2026Spectral-Progressive Thought Flow for Lightweight Multimodal Reasoningcited by 1
  3. 2026DCP-Prune: Ultra-Low Token Pruning with Distribution Consistency Preservationcited by 0
  4. 2026Buried in Textual Debt: Context Pruning with Visual Evidence Preservation for MLLM Agentscited by 0
  5. 2026VisCache: Visual KV Cache Pruning for Efficient Vision Large Language Model Inferencecited by 0
  6. 2026Doc-REFRAG: Rethinking Multimodal Document Retrieval-Augmented Generationcited by 0
  7. 2026DecoMoE: Decoupling Visual Propagation and Expert Computation for Efficient Multimodal MoE Inferencecited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025VisionSelector: End-to-End Learnable Visual Token Compression for Efficient Multimodal LLMs
  2. 2025Are We Using the Right Benchmark: An Evaluation Framework for Visual Token Compression Methods
  3. 2026Stepwise Token Selection for Efficient Multimodal Large Language Models
  4. 2024FocusLLaVA: A Coarse-to-Fine Approach for Efficient and Effective Visual Token Compression
  5. 2025Token Sequence Compression for Efficient Multimodal Computing
  6. 2026EvoComp: Learning Visual Token Compression for Multimodal Large Language Models via Semantic-Guided Evolutionary Labeling
  7. 2024ST3: Accelerating Multimodal Large Language Model by Spatial-Temporal Visual Token Trimming
  8. 2024Filter, Correlate, Compress: Training-Free Token Reduction for MLLM Acceleration