# AutoPrune: Each Complexity Deserves a Pruning Policy

arXiv: https://arxiv.org/abs/2509.23931 · HTML: https://mlsanity.com/paper/2509.23931/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- Placed in Visual token pruning and efficiency, one of 16 directions in Multimodal LLMs / vision-language models.
- Builds on 31 of its 51 references that are in the map.
- First posted on arXiv 28 Sep 2025.

## Placed in

- Multimodal LLMs / vision-language models / [Visual token pruning and efficiency](https://mlsanity.com/mllm/visual-token-pruning-and-efficiency/)

## Builds on (most cited first, 30 of 31)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Visual Instruction Tuning](https://mlsanity.com/paper/2304.08485/) (2023)
- [Representation Learning with Contrastive Predictive Coding](https://mlsanity.com/paper/1807.03748/) (2018)
- [BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Models](https://mlsanity.com/paper/2301.12597/) (2023)
- [BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generation](https://mlsanity.com/paper/2201.12086/) (2022)
- [OpenVLA: An Open-Source Vision-Language-Action Model](https://mlsanity.com/paper/2406.09246/) (2024)
- [VQA: Visual Question Answering](https://mlsanity.com/paper/1505.00468/) (2015)
- [π0: A Vision-Language-Action Flow Model for General Robot Control](https://mlsanity.com/paper/2410.24164/) (2024)
- [Evaluating Object Hallucination in Large Vision-Language Models](https://mlsanity.com/paper/2305.10355/) (2023)
- [Towards VQA Models That Can Read](https://mlsanity.com/paper/1904.08920/) (2019)
- [Microsoft COCO Captions: Data Collection and Evaluation Server](https://mlsanity.com/paper/1504.00325/) (2015)
- [mPLUG-Owl: Modularization Empowers Large Language Models with Multimodality](https://mlsanity.com/paper/2304.14178/) (2023)
- [Token Merging: Your ViT But Faster](https://mlsanity.com/paper/2210.09461/) (2022)
- [An Image is Worth 1/2 Tokens After Layer 2: Plug-and-Play Inference Acceleration for Large Vision-Language Models](https://mlsanity.com/paper/2403.06764/) (2024)
- [GIT: A Generative Image-to-text Transformer for Vision and Language](https://mlsanity.com/paper/2205.14100/) (2022)
- [SparseVLM: Visual Token Sparsification for Efficient Vision-Language Model Inference](https://mlsanity.com/paper/2410.04417/) (2024)
- [VisionZip: Longer is Better but Not Necessary in Vision Language Models](https://mlsanity.com/paper/2412.04467/) (2024)
- [PyramidDrop: Accelerating Your Large Vision-Language Models via Pyramid Visual Redundancy Reduction](https://mlsanity.com/paper/2410.17247/) (2024)
- [Vision-Language Pre-Training with Triple Contrastive Learning](https://mlsanity.com/paper/2202.10401/) (2022)
- [BlenderBot 3: a deployed conversational agent that continually learns to responsibly engage](https://mlsanity.com/paper/2208.03188/) (2022)
- [Senna: Bridging Large Vision-Language Models and End-to-End Autonomous Driving](https://mlsanity.com/paper/2410.22313/) (2024)
- [Token Fusion: Bridging the Gap between Token Pruning and Token Merging](https://mlsanity.com/paper/2312.01026/) (2023)
- [A General Survey on Attention Mechanisms in Deep Learning](https://mlsanity.com/paper/2203.14263/) (2022)
- [Dynamic Token Pruning in Plain Vision Transformers for Semantic Segmentation](https://mlsanity.com/paper/2308.01045/) (2023)
- [TopV: Compatible Token Pruning with Inference Time Optimization for Fast and Low-Memory Multimodal Vision Language Model](https://mlsanity.com/paper/2503.18278/) (2025)
- [ZipVL: Efficient Large Vision-Language Models with Dynamic Token Sparsification](https://mlsanity.com/paper/2410.08584/) (2024)
- [Adaptive Sparse ViT: Towards Learnable Adaptive Token Pruning by Fully Exploiting Self-Attention](https://mlsanity.com/paper/2209.13802/) (2022)
- [Revealing Vision-Language Integration in the Brain with Multimodal Networks](https://mlsanity.com/paper/2406.14481/) (2024)
- [VLTP: Vision-Language Guided Token Pruning for Task-Oriented Segmentation](https://mlsanity.com/paper/2409.08464/) (2024)
- [Video Token Sparsification for Efficient Multimodal LLMs in Autonomous Driving](https://mlsanity.com/paper/2409.11182/) (2024)

## Built on by (most cited first, 4 of 4)

- [QMoP: Query Guided Mixture-of-Projector for Efficient Visual Token Compression](https://mlsanity.com/paper/2603.21232/) (2026)
- [Structural Pruning of Large Vision Language Models: A Comprehensive Study on Pruning Dynamics, Recovery, and Data Efficiency](https://mlsanity.com/paper/2604.24380/) (2026)
- [SinkPruner: Sink-Free Visual Token Pruning for Multimodal Large Language Models](https://mlsanity.com/paper/2609.01004/) (2026)
- [MI-Pruner: Crossmodal Mutual Information-guided Token Pruner for Efficient MLLMs](https://mlsanity.com/paper/2604.03072/) (2026)

## Abstract (arXiv)

The established redundancy in visual tokens within large vision-language models allows pruning to effectively reduce their substantial computational demands. Previous methods typically employ heuristic layer-specific pruning strategies where, although the number of tokens removed may differ across decoder layers, the overall pruning schedule is fixed and applied uniformly to all input samples and tasks, failing to align token elimination with the model's holistic reasoning trajectory. Cognitive science indicates that human visual processing often begins with broad exploration to accumulate evidence before narrowing focus as the target becomes distinct. Our experiments reveal an analogous pattern in these models. This observation suggests that neither a fixed pruning schedule nor a heuristic layer-wise strategy can optimally accommodate the diverse complexities inherent in different inputs. To overcome this limitation, we introduce Complexity-Adaptive Pruning (AutoPrune), a training-free, plug-and-play framework that tailors pruning policies to varying sample and task complexities. Specifically, AutoPrune quantifies the mutual information between visual and textual tokens, then projects this signal to a budget-constrained logistic retention curve. Each such logistic curve, defined by its unique shape, corresponds to the specific complexity of different tasks and can guarantee adherence to predefined computational constraints. We evaluate AutoPrune on standard vision-language tasks and on Vision-Language-Action models for autonomous driving. Notably, when applied to LLaVA-1.5-7B, our method prunes 89% of visual tokens and reduces inference FLOPs by 76.8% while retaining 96.7% of the original accuracy averaged over all tasks. This corresponds to a 9.1% improvement over the recent work PDrop, demonstrating the effectiveness. Code is available at https://github.com/AutoLab-SAI-SJTU/AutoPrune.