# Pruning All-Rounder: Rethinking and Improving Inference Efficiency for Large Vision Language Models

arXiv: https://arxiv.org/abs/2412.06458 · HTML: https://mlsanity.com/paper/2412.06458/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- Placed in Multimodal reasoning, one of 17 directions in LLM reasoning.
- Builds on 9 of its 52 references that are in the map.
- First posted on arXiv 9 Dec 2024.

## Placed in

- LLM reasoning / [Multimodal reasoning](https://mlsanity.com/lr/multimodal-reasoning/)
- Multimodal LLMs / vision-language models / [Visual token pruning and efficiency](https://mlsanity.com/mllm/visual-token-pruning-and-efficiency/)

## Builds on (most cited first, 9 of 9)

- [Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedback](https://mlsanity.com/paper/2204.05862/) (2022)
- [Evaluating Object Hallucination in Large Vision-Language Models](https://mlsanity.com/paper/2305.10355/) (2023)
- [SimPO: Simple Preference Optimization with a Reference-Free Reward](https://mlsanity.com/paper/2405.14734/) (2024)
- [Aligning Large Multimodal Models with Factually Augmented RLHF](https://mlsanity.com/paper/2309.14525/) (2023)
- [ORPO: Monolithic Preference Optimization without Reference Model](https://mlsanity.com/paper/2403.07691/) (2024)
- [Silkie: Preference Distillation for Large Visual Language Models](https://mlsanity.com/paper/2312.10665/) (2023)
- [Fit and Prune: Fast and Training-free Visual Token Pruning for Multi-modal Large Language Models](https://mlsanity.com/paper/2409.10197/) (2024)
- [γ-MoD: Exploring Mixture-of-Depth Adaptation for Multimodal Large Language Models](https://mlsanity.com/paper/2410.13859/) (2024)
- [Patch Ranking: Efficient CLIP by Learning to Rank Local Patches](https://mlsanity.com/paper/2409.14607/) (2024)

## Built on by (most cited first, 4 of 4)

- [AVERY: Intent-Driven Adaptive VLM Split Computing via Embodied Self-Awareness for Efficient Disaster Response Systems](https://mlsanity.com/paper/2511.18151/) (2025)
- [Do Vision Language Models Need to Process Image Tokens?](https://mlsanity.com/paper/2604.09425/) (2026)
- [Hallucination-aware intermediate representation edit in large vision-language models](https://mlsanity.com/paper/2603.29405/) (2026)
- [SPIDER: Multi-Layer Semantic Token Pruning and Adaptive Sub-Layer Skipping in Multimodal Large Language Models](https://mlsanity.com/paper/2609.34977/) (2026)

## Abstract (arXiv)

Although Large Vision-Language Models (LVLMs) have achieved impressive results, their high computational costs pose a significant barrier to wide application. To enhance inference efficiency, most existing approaches can be categorized as parameter-dependent or token-dependent strategies to reduce computational demands. However, parameter-dependent methods require retraining LVLMs to recover performance while token-dependent strategies struggle to consistently select the most relevant tokens. In this paper, we systematically analyze the above challenges and provide a series of valuable insights for inference acceleration. Based on these findings, we propose a novel framework, the Pruning All-Rounder (PAR). Different from previous works, PAR develops a meta-router to adaptively organize pruning flows across both tokens and layers. With a self-supervised learning manner, our method achieves a superior balance between performance and efficiency. Notably, PAR is highly flexible, offering multiple pruning versions to address a range of acceleration scenarios. The code for this work is publicly available at https://github.com/ASGO-MM/Pruning-All-Rounder.