mlsanityData to 14 Oct 2026

Efficient LLaMA-3.2-Vision by Trimming Cross-attended Visual Features

arXiv 2504.00557 ↗First posted 1 Apr 2025cs.CV cs.LGcited by 10

On the map: Multimodal LLMs / vision-language models → Visual token pruning and efficiency

also in 1 more
Abstract (arXiv)

Visual token reduction lowers inference costs caused by extensive image features in large vision-language models (LVLMs). Unlike relevant studies that prune tokens in self-attention-only LVLMs, our work uniquely addresses cross-attention-based models, which achieve superior performance. We identify that the key-value (KV) cache size for image tokens in cross-attention layers significantly exceeds that of text tokens in self-attention layers, posing a major compute bottleneck. To mitigate this issue, we exploit the sparse nature in cross-attention maps to selectively prune redundant visual features. Our Trimmed Llama effectively reduces KV cache demands without requiring additional training. By benefiting from 50%-reduced visual features, our model can reduce inference latency and memory usage while achieving benchmark parity.

Key facts. Cited by 10 papers in the map (data through 14 Oct 2026). Placed in Visual token pruning and efficiency, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 26 of its 27 references that are in the map. First posted on arXiv 1 Apr 2025.

Builds on

26 of 27 references.

Closest ancestors · 7

Same direction, newest first

  1. 2024ZipVL: Efficient Large Vision-Language Models with Dynamic Token Sparsification11 Oct
  2. 2024Efficient Inference of Vision Instruction-Following Models with Elastic Cache25 Jul
  3. 2024LOOK-M: Look-Once Optimization in KV Cache for Efficient Multimodal Long-Context Inference26 Jun
  4. 2024Matryoshka Multimodal Models27 May
  5. 2024LLaVA-PruMerge: Adaptive Token Reduction for Efficient Large Multimodal Models22 Mar
  6. 2024LLaVA-UHD: an LMM Perceiving Any Aspect Ratio and High-Resolution Images18 Mar
  7. 2024An Image is Worth 1/2 Tokens After Layer 2: Plug-and-Play Inference Acceleration for Large Vision-Language Models11 Mar

Other references · 19

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  2. 2024The Llama 3 Herd of Modelscited by 14,715
  3. 2023Visual Instruction Tuningcited by 7,869
  4. 2022Flamingo: a Visual Language Model for Few-Shot Learningcited by 4,386
  5. 2024LLaVA-OneVision: Easy Visual Task Transfercited by 2,719
  6. 2023Efficient Streaming Language Models with Attention Sinkscited by 2,186
  7. 2023MME: A Comprehensive Evaluation Benchmark for Multimodal Large Language Modelscited by 1,652
  8. 2024Expanding Performance Boundaries of Open-Source Multimodal Models with Model, Data, and Test-Time Scalingcited by 1,599
  9. 2023SEED-Bench: Benchmarking Multimodal LLMs with Generative Comprehensioncited by 940
  10. 2024SnapKV: LLM Knows What You are Looking for Before Generationcited by 880
  11. 2023H2O: Heavy-Hitter Oracle for Efficient Generative Inference of Large Language Modelscited by 832
  12. 2024Eyes Wide Shut? Exploring the Visual Shortcomings of Multimodal LLMscited by 735
7 more
  1. 2023VILA: On Pre-training for Visual Language Modelscited by 692
  2. 2023OpenFlamingo: An Open-Source Framework for Training Large Autoregressive Vision-Language Modelscited by 565
  3. 2023Model Tells You What to Discard: Adaptive KV Cache Compression for LLMscited by 474
  4. 2024PyramidKV: Dynamic KV Cache Compression based on Pyramidal Information Funnelingcited by 432
  5. 2024LMMs-Eval: Reality Check on the Evaluation of Large Multimodal Modelscited by 353
  6. 2023MIMIC-IT: Multi-Modal In-Context Instruction Tuningcited by 267
  7. 2024NVLM: Open Frontier-Class Multimodal LLMscited by 125

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 5; 2026: 5.

10 papers.

  1. 2025ChartM3: Benchmarking Chart Editing with Multimodal Instructionscited by 9
  2. 2025IF-VidCap: Can Video Caption Models Follow Instructions?cited by 7
  3. 2025ERGO: Efficient High-Resolution Visual Understanding for Vision-Language Modelscited by 4
  4. 2026HotComment: A Benchmark for Evaluating Popularity of Online Commentscited by 3
  5. 2026C-CoT: Counterfactual Chain-of-Thought with Vision-Language Models for Safe Autonomous Drivingcited by 3
  6. 2026Focal-RegionFace: Generating Fine-Grained Multi-attribute Descriptions for Arbitrarily Selected Face Focal Regionscited by 1
  7. 2025Robotic Fire Risk Detection based on Dynamic Knowledge Graph Reasoning: An LLM-Driven Approach with Graph Chain-of-Thoughtcited by 0
  8. 2025Towards General Computer Control with Hierarchical Agents and Multi-Level Action Spacescited by 0
  9. 2026Unveiling Covert Toxicity in Multimodal Data via Toxicity Association Graphs: A Graph-Based Metric and Interpretable Detection Frameworkcited by 0
  10. 2026MMTClinic: Multimodal, Multilingual Time Series Question Answering and Reasoning Benchmark for Clinical Domaincited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026Efficient Inference for Large Vision-Language Models: Bottlenecks, Techniques, and Prospects
  2. 2026ASAP: Attention-Shift-Aware Pruning for Efficient LVLM Inference
  3. 2026ACPruner: Visual Token Pruning as Biased Attention Coverage Maximization in LVLMs
  4. 2025VScan: Rethinking Visual Token Reduction for Efficient Large Vision-Language Models
  5. 2026Attention-Free and Lightweight Token Reduction for Efficient Vision-Language Models
  6. 2026CRISP: Pre-LLM Yet Text-Driven Visual Token Pruning for Efficient LVLM Inference
  7. 2025AirCache: Activating Inter-modal Relevancy KV Cache Compression for Efficient Large Vision-Language Model Inference
  8. 2024A Stitch in Time Saves Nine: Small VLM is a Precise Guidance for Accelerating Large VLMs