# CAPA: Contribution-Aware Pruning and FFN Approximation for Efficient Large Vision-Language Models

arXiv: https://arxiv.org/abs/2602.00247 · HTML: https://mlsanity.com/paper/2602.00247/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Placed in Visual token pruning and efficiency, one of 16 directions in Multimodal LLMs / vision-language models.
- Builds on 23 of its 37 references that are in the map.
- First posted on arXiv 30 Jan 2026.

## Placed in

- Multimodal LLMs / vision-language models / [Visual token pruning and efficiency](https://mlsanity.com/mllm/visual-token-pruning-and-efficiency/)

## Builds on (most cited first, 23 of 23)

- [Microsoft COCO: Common Objects in Context](https://mlsanity.com/paper/1405.0312/) (2014)
- [Qwen2-VL: Enhancing Vision-Language Model's Perception of the World at Any Resolution](https://mlsanity.com/paper/2409.12191/) (2024)
- [Making the V in VQA Matter: Elevating the Role of Image Understanding in Visual Question Answering](https://mlsanity.com/paper/1612.00837/) (2016)
- [Efficient Streaming Language Models with Attention Sinks](https://mlsanity.com/paper/2309.17453/) (2023)
- [MMMU: A Massive Multi-discipline Multimodal Understanding and Reasoning Benchmark for Expert AGI](https://mlsanity.com/paper/2311.16502/) (2023)
- [Towards VQA Models That Can Read](https://mlsanity.com/paper/1904.08920/) (2019)
- [MM-Vet: Evaluating Large Multimodal Models for Integrated Capabilities](https://mlsanity.com/paper/2308.02490/) (2023)
- [SparseGPT: Massive Language Models Can Be Accurately Pruned in One-Shot](https://mlsanity.com/paper/2301.00774/) (2023)
- [SEED-Bench: Benchmarking Multimodal LLMs with Generative Comprehension](https://mlsanity.com/paper/2307.16125/) (2023)
- [SVD-LLM: Truncation-aware Singular Value Decomposition for Large Language Model Compression](https://mlsanity.com/paper/2403.07378/) (2024)
- [MegaBlocks: Efficient Sparse Training with Mixture-of-Experts](https://mlsanity.com/paper/2211.15841/) (2022)
- [Beyond Text-Visual Attention: Exploiting Visual Cues for Effective Token Pruning in VLMs](https://mlsanity.com/paper/2412.01818/) (2024)
- [MoEfication: Transformer Feed-forward Layers are Mixtures of Experts](https://mlsanity.com/paper/2110.01786/) (2021)
- [Attention Score is not All You Need for Token Importance Indicator in KV Cache Reduction: Value Also Matters](https://mlsanity.com/paper/2406.12335/) (2024)
- [Monarch Mixer: A Simple Sub-Quadratic GEMM-Based Architecture](https://mlsanity.com/paper/2310.12109/) (2023)
- [Attention Sinks and Compression Valleys in LLMs are Two Sides of the Same Coin](https://mlsanity.com/paper/2510.06477/) (2025)
- [Feather the Throttle: Revisiting Visual Token Pruning for Vision-Language Model Acceleration](https://mlsanity.com/paper/2412.13180/) (2024)
- [Treat Visual Tokens as Text? But Your MLLM Only Needs Fewer Efforts to See](https://mlsanity.com/paper/2410.06169/) (2024)
- [TensorGPT: Efficient Compression of Large Language Models based on Tensor-Train Decomposition](https://mlsanity.com/paper/2307.00526/) (2023)
- [How Multimodal LLMs Solve Image Tasks: A Lens on Visual Grounding, Task Reasoning, and Answer Decoding](https://mlsanity.com/paper/2508.20279/) (2025)
- [FREE: Fast and Robust Vision Language Models with Early Exits](https://mlsanity.com/paper/2506.06884/) (2025)
- [RedundancyLens: Revealing and Exploiting Visual Token Processing Redundancy for Efficient Decoder-Only MLLMs](https://mlsanity.com/paper/2501.19036/) (2025)
- [Surgical Feature-Space Decomposition of LLMs: Why, When and How?](https://mlsanity.com/paper/2405.13039/) (2024)

## Built on by (most cited first, 1 of 1)

- [From Saliency to Discriminability: Rank-Preserving Visual Token Pruning for VLM Rerankers](https://mlsanity.com/paper/2609.00667/) (2026)

## Abstract (arXiv)

Efficient inference in Large Vision-Language Models is constrained by the high cost of processing thousands of visual tokens, yet it remains unclear which tokens and computations can be safely removed. While attention scores are commonly used to estimate visual token importance, they are an imperfect proxy for actual contribution. We show that Attention Contribution, which weights attention probabilities by value vector magnitude, provides a more accurate criterion for visual token selection. Our empirical analysis reveals that visual attention sinks are functionally heterogeneous, comprising Probability Dumps with low contribution that can be safely pruned, and Structural Anchors with high contribution essential for maintaining model performance. Further, we identify substantial redundancy in Feed-Forward Networks (FFNs) associated with visual tokens, particularly in intermediate layers where image tokens exhibit linear behavior. Based on our findings, we introduce CAPA (Contribution-Aware Pruning and FFN Approximation), a dual-strategy framework that prunes visual tokens using attention contribution at critical functional transitions and reduces FFN computation through efficient linear approximations. Experiments on various benchmarks across baselines show that CAPA achieves competent efficiency--performance trade-offs with improved robustness.