# TAMP: Token-Adaptive Layerwise Pruning in Multimodal Large Language Models

arXiv: https://arxiv.org/abs/2504.09897 · HTML: https://mlsanity.com/paper/2504.09897/

## Key facts

- Cited by 9 papers in the map (data through 14 Oct 2026).
- Placed in Visual token pruning and efficiency, one of 16 directions in Multimodal LLMs / vision-language models.
- Builds on 35 of its 50 references that are in the map.
- First posted on arXiv 14 Apr 2025.

## Placed in

- Multimodal LLMs / vision-language models / [Visual token pruning and efficiency](https://mlsanity.com/mllm/visual-token-pruning-and-efficiency/)

## Builds on (most cited first, 30 of 35)

- [LLaMA: Open and Efficient Foundation Language Models](https://mlsanity.com/paper/2302.13971/) (2023)
- [Llama 2: Open Foundation and Fine-Tuned Chat Models](https://mlsanity.com/paper/2307.09288/) (2023)
- [Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context](https://mlsanity.com/paper/2403.05530/) (2024)
- [LLaVA-OneVision: Easy Visual Task Transfer](https://mlsanity.com/paper/2408.03326/) (2024)
- [MMBench: Is Your Multi-modal Model an All-around Player?](https://mlsanity.com/paper/2307.06281/) (2023)
- [MMMU: A Massive Multi-discipline Multimodal Understanding and Reasoning Benchmark for Expert AGI](https://mlsanity.com/paper/2311.16502/) (2023)
- [GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformers](https://mlsanity.com/paper/2210.17323/) (2022)
- [MathVista: Evaluating Mathematical Reasoning of Foundation Models in Visual Contexts](https://mlsanity.com/paper/2310.02255/) (2023)
- [ChartQA: A Benchmark for Question Answering about Charts with Visual and Logical Reasoning](https://mlsanity.com/paper/2203.10244/) (2022)
- [AWQ: Activation-aware Weight Quantization for LLM Compression and Acceleration](https://mlsanity.com/paper/2306.00978/) (2023)
- [SparseGPT: Massive Language Models Can Be Accurately Pruned in One-Shot](https://mlsanity.com/paper/2301.00774/) (2023)
- [ShareGPT4V: Improving Large Multi-Modal Models with Better Captions](https://mlsanity.com/paper/2311.12793/) (2023)
- [Cambrian-1: A Fully Open, Vision-Centric Exploration of Multimodal LLMs](https://mlsanity.com/paper/2406.16860/) (2024)
- [A Diagram Is Worth A Dozen Images](https://mlsanity.com/paper/1603.07396/) (2016)
- [VideoLLaMA 2: Advancing Spatial-Temporal Modeling and Audio Understanding in Video-LLMs](https://mlsanity.com/paper/2406.07476/) (2024)
- [NExT-GPT: Any-to-Any Multimodal LLM](https://mlsanity.com/paper/2309.05519/) (2023)
- [To prune, or not to prune: exploring the efficacy of pruning for model compression](https://mlsanity.com/paper/1710.01878/) (2017)
- [ZeroQuant: Efficient and Affordable Post-Training Quantization for Large-Scale Transformers](https://mlsanity.com/paper/2206.01861/) (2022)
- [A Corpus for Reasoning About Natural Language Grounded in Photographs](https://mlsanity.com/paper/1811.00491/) (2018)
- [ShortGPT: Layers in Large Language Models are More Redundant Than You Expect](https://mlsanity.com/paper/2403.03853/) (2024)
- [AnyGPT: Unified Multimodal LLM with Discrete Sequence Modeling](https://mlsanity.com/paper/2402.12226/) (2024)
- [MANTIS: Interleaved Multi-Image Instruction Tuning](https://mlsanity.com/paper/2405.01483/) (2024)
- [Learning to Answer Questions in Dynamic Audio-Visual Scenarios](https://mlsanity.com/paper/2203.14072/) (2022)
- [MuirBench: A Comprehensive Benchmark for Robust Multi-image Understanding](https://mlsanity.com/paper/2406.09411/) (2024)
- [Learning N:M Fine-grained Structured Sparse Neural Networks From Scratch](https://mlsanity.com/paper/2102.04010/) (2021)
- [Outlier Weighed Layerwise Sparsity (OWL): A Missing Secret Sauce for Pruning LLMs to High Sparsity](https://mlsanity.com/paper/2310.05175/) (2023)
- [Clotho-AQA: A Crowdsourced Dataset for Audio Question Answering](https://mlsanity.com/paper/2204.09634/) (2022)
- [D2 Pruning: Message Passing for Balancing Diversity and Difficulty in Data Pruning](https://mlsanity.com/paper/2310.07931/) (2023)
- [X-Pruner: eXplainable Pruning for Vision Transformers](https://mlsanity.com/paper/2303.04935/) (2023)
- [LoRAPrune: Structured Pruning Meets Low-Rank Parameter-Efficient Fine-Tuning](https://mlsanity.com/paper/2305.18403/) (2023)

## Built on by (most cited first, 9 of 9)

- [Get Experience from Practice: LLM Agents with Record & Replay](https://mlsanity.com/paper/2505.17716/) (2025)
- [UniVLR: Unifying Text and Vision in Visual Latent Reasoning for Multimodal LLMs](https://mlsanity.com/paper/2605.11856/) (2026)
- [ReGATE: Learning Faster and Better with Fewer Tokens in MLLMs](https://mlsanity.com/paper/2507.21420/) (2025)
- [MuCRASP: Multimodal Chain-of-thought Reasoning aware Structured Pruning](https://mlsanity.com/paper/2605.25842/) (2026)
- [A Vision for Multisensory Intelligence: Sensing, Science, and Synergy](https://mlsanity.com/paper/2601.04563/) (2026)
- [Mostly Text, Smart Visuals: Asymmetric Text-Visual Pruning for Large Vision-Language Models](https://mlsanity.com/paper/2603.16001/) (2026)
- [Attention-Free and Lightweight Token Reduction for Efficient Vision-Language Models](https://mlsanity.com/paper/2607.13500/) (2026)
- [Buried in Textual Debt: Context Pruning with Visual Evidence Preservation for MLLM Agents](https://mlsanity.com/paper/2608.22963/) (2026)
- [When and What to Prune? Stage-Aware Visual Token Pruning for Efficient VLA](https://mlsanity.com/paper/2610.05273/) (2026)

## Abstract (arXiv)

Multimodal Large Language Models (MLLMs) have shown remarkable versatility in understanding diverse multimodal data and tasks. However, these capabilities come with an increased model scale. While post-training pruning reduces model size in unimodal models, its application to MLLMs often yields limited success. Our analysis discovers that conventional methods fail to account for the unique token attributes across layers and modalities inherent to MLLMs. Inspired by this observation, we propose TAMP, a simple yet effective pruning framework tailored for MLLMs, featuring two key components: (1) Diversity-Aware Sparsity, which adjusts sparsity ratio per layer based on diversities among multimodal output tokens, preserving more parameters in high-diversity layers; and (2) Adaptive Multimodal Input Activation, which identifies representative multimodal input tokens using attention scores to guide unstructured weight pruning. We validate our method on two state-of-the-art MLLMs: LLaVA-NeXT, designed for vision-language tasks, and VideoLLaMA2, capable of processing audio, visual, and language modalities. Empirical experiments across various multimodal evaluation benchmarks demonstrate that each component of our approach substantially outperforms existing pruning techniques.