# CATP: Contextually Adaptive Token Pruning for Efficient and Enhanced Multimodal In-Context Learning

arXiv: https://arxiv.org/abs/2508.07871 · HTML: https://mlsanity.com/paper/2508.07871/

## Key facts

- Cited by 18 papers in the map (data through 14 Oct 2026).
- Placed in Visual token pruning and efficiency, one of 16 directions in Multimodal LLMs / vision-language models.
- Builds on 29 of its 44 references that are in the map.
- First posted on arXiv 11 Aug 2025.

## Placed in

- Multimodal LLMs / vision-language models / [Visual token pruning and efficiency](https://mlsanity.com/mllm/visual-token-pruning-and-efficiency/)
- Efficient LLM inference and serving / [Visual token pruning in multimodal models](https://mlsanity.com/inference/visual-token-pruning-in-multimodal-models/)

## Builds on (most cited first, 29 of 29)

- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [Qwen2-VL: Enhancing Vision-Language Model's Perception of the World at Any Resolution](https://mlsanity.com/paper/2409.12191/) (2024)
- [FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness](https://mlsanity.com/paper/2205.14135/) (2022)
- [LLaVA-OneVision: Easy Visual Task Transfer](https://mlsanity.com/paper/2408.03326/) (2024)
- [InternVL: Scaling up Vision Foundation Models and Aligning for Generic Visual-Linguistic Tasks](https://mlsanity.com/paper/2312.14238/) (2023)
- [Making the V in VQA Matter: Elevating the Role of Image Understanding in Visual Question Answering](https://mlsanity.com/paper/1612.00837/) (2016)
- [Making Pre-trained Language Models Better Few-shot Learners](https://mlsanity.com/paper/2012.15723/) (2020)
- [What Makes Good In-Context Examples for GPT-3?](https://mlsanity.com/paper/2101.06804/) (2021)
- [OK-VQA: A Visual Question Answering Benchmark Requiring External Knowledge](https://mlsanity.com/paper/1906.00067/) (2019)
- [VizWiz Grand Challenge: Answering Visual Questions from Blind People](https://mlsanity.com/paper/1802.08218/) (2018)
- [Large Language Models](https://mlsanity.com/paper/2307.05782/) (2023)
- [A Survey on In-context Learning](https://mlsanity.com/paper/2301.00234/) (2022)
- [The Hateful Memes Challenge: Detecting Hate Speech in Multimodal Memes](https://mlsanity.com/paper/2005.04790/) (2020)
- [SparseVLM: Visual Token Sparsification for Efficient Vision-Language Model Inference](https://mlsanity.com/paper/2410.04417/) (2024)
- [Monkey: Image Resolution and Text Label Are Important Things for Large Multi-modal Models](https://mlsanity.com/paper/2311.06607/) (2023)
- [PyramidDrop: Accelerating Your Large Vision-Language Models via Pyramid Visual Redundancy Reduction](https://mlsanity.com/paper/2410.17247/) (2024)
- [MME-RealWorld: Could Your Multimodal LLM Challenge High-Resolution Real-World Scenarios that are Difficult for Humans?](https://mlsanity.com/paper/2408.13257/) (2024)
- [Stop Looking for Important Tokens in Multimodal Language Models: Duplication Matters More](https://mlsanity.com/paper/2502.11494/) (2025)
- [Boosting Multimodal Large Language Models with Visual Tokens Withdrawal for Rapid Inference](https://mlsanity.com/paper/2405.05803/) (2024)
- [Beyond Attention or Similarity: Maximizing Conditional Diversity for Token Pruning in MLLMs](https://mlsanity.com/paper/2506.10967/) (2025)
- [Token Pruning in Multimodal Large Language Models: Are We Solving the Right Problem?](https://mlsanity.com/paper/2502.11501/) (2025)
- [What Makes Multimodal In-Context Learning Work?](https://mlsanity.com/paper/2404.15736/) (2024)
- [VL-ICL Bench: The Devil in the Details of Multimodal In-Context Learning](https://mlsanity.com/paper/2403.13164/) (2024)
- [Can Multimodal Large Language Models Truly Perform Multimodal In-Context Learning?](https://mlsanity.com/paper/2311.18021/) (2023)
- [TACO: Enhancing Multimodal In-context Learning via Task Mapping-Guided Sequence Configuration](https://mlsanity.com/paper/2505.17098/) (2025)
- [ParallelComp: Parallel Long-Context Compressor for Length Extrapolation](https://mlsanity.com/paper/2502.14317/) (2025)
- [LF-ViT: Reducing Spatial Redundancy in Vision Transformer for Efficient Image Recognition](https://mlsanity.com/paper/2402.00033/) (2024)
- [VEGA: Learning Interleaved Image-Text Comprehension in Vision-Language Large Models](https://mlsanity.com/paper/2406.10228/) (2024)
- [Understanding Attention for Vision-and-Language Tasks](https://mlsanity.com/paper/2208.08104/) (2022)

## Built on by (most cited first, 18 of 18)

- [Look Inward to Explore Outward: Learning Temperature Policy from LLM Internal States via Hierarchical RL](https://mlsanity.com/paper/2602.13035/) (2026)
- [Causal Tracing of Object Representations in Large Vision Language Models: Mechanistic Interpretability and Hallucination Mitigation](https://mlsanity.com/paper/2511.05923/) (2025)
- [TrimTokenator-LC: Towards Adaptive Visual Token Pruning for Large Multimodal Models with Long Contexts](https://mlsanity.com/paper/2512.22748/) (2025)
- [MedSynapse-V: Bridging Visual Perception and Clinical Intuition via Latent Memory Evolution](https://mlsanity.com/paper/2604.26283/) (2026)
- [Hy-Facial: Hybrid Feature Extraction by Dimensionality Reduction Methods for Enhanced Facial Expression Classification](https://mlsanity.com/paper/2509.26614/) (2025)
- [ChartVerse: Scaling Chart Reasoning via Reliable Programmatic Synthesis from Scratch](https://mlsanity.com/paper/2601.13606/) (2026)
- [TrimTokenator: Towards Adaptive Visual Token Pruning for Large Multimodal Models](https://mlsanity.com/paper/2509.00320/) (2025)
- [When Models Learn to Ask Why: Adaptive Causal Reasoning for Trustworthy Medical Vision-Language Models](https://mlsanity.com/paper/2603.23085/) (2026)
- [EvoStreaming: Your Offline Video Model Is a Natively Streaming Assistant](https://mlsanity.com/paper/2605.10343/) (2026)
- [AVIS: Adaptive Test-Time Scaling for Vision-Language Models](https://mlsanity.com/paper/2606.11576/) (2026)
- [Improving Alignment in LVLMs with Debiased Self-Judgment](https://mlsanity.com/paper/2508.20655/) (2025)
- [Personalize Your Large Vision-language Models With In-context Prompt Tuning](https://mlsanity.com/paper/2605.31513/) (2026)
- [Why Multimodal In-Context Learning Lags Behind? Unveiling the Inner Mechanisms and Bottlenecks](https://mlsanity.com/paper/2604.13403/) (2026)
- [Combating Textual Noise and Redundancy: Entropy-Aware Dense Visual Token Pruning](https://mlsanity.com/paper/2607.02484/) (2026)
- [A Systematic Study of Model Extraction Attacks on Graph Foundation Models](https://mlsanity.com/paper/2511.11912/) (2025)
- [UniCreative: Unifying Long-form Logic and Short-form Sparkle via Reference-Free Reinforcement Learning](https://mlsanity.com/paper/2604.05517/) (2026)
- [Exploiting answer-invariant redundancies in satellite imagery for efficient VLM inference on edge](https://mlsanity.com/paper/2609.29029/) (2026)
- [MCD: Causal Distillation of Multimodal In-Context Learning in Large Vision-Language Models](https://mlsanity.com/paper/2609.39920/) (2026)

## Abstract (arXiv)

Modern large vision-language models (LVLMs) convert each input image into a large set of tokens that far outnumber the text tokens. Although this improves visual perception, it also introduces severe image token redundancy. Because image tokens contain sparse information, many contribute little to reasoning but greatly increase inference cost. Recent image token pruning methods address this issue by identifying important tokens and removing the rest. These methods improve efficiency with only small performance drops. However, most of them focus on single-image tasks and overlook multimodal in-context learning (ICL), where redundancy is higher and efficiency is more important. Redundant tokens weaken the advantage of multimodal ICL for rapid domain adaptation and lead to unstable performance. When existing pruning methods are applied in this setting, they cause large accuracy drops, which exposes a clear gap and the need for new approaches. To address this, we propose Contextually Adaptive Token Pruning (CATP), a training-free pruning method designed for multimodal ICL. CATP uses two stages of progressive pruning that fully reflect the complex cross-modal interactions in the input sequence. After removing 77.8% of the image tokens, CATP achieves an average performance gain of 0.6% over the vanilla model on four LVLMs and eight benchmarks, clearly outperforming all baselines. At the same time, it improves efficiency by reducing inference latency by an average of 10.78%. CATP strengthens the practical value of multimodal ICL and lays the foundation for future progress in interleaved image-text settings.