# P4Q: Co-designing Token Pruning and Quantization for Vision-Language Model Acceleration

arXiv: https://arxiv.org/abs/2609.34867 · HTML: https://mlsanity.com/paper/2609.34867/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Placed in Visual token pruning and efficiency, one of 16 directions in Multimodal LLMs / vision-language models.
- Builds on 27 of its 30 references that are in the map.
- First posted on arXiv 28 Sep 2026.

## Placed in

- Multimodal LLMs / vision-language models / [Visual token pruning and efficiency](https://mlsanity.com/mllm/visual-token-pruning-and-efficiency/)
- Efficient LLM inference and serving / [Visual token pruning in multimodal models](https://mlsanity.com/inference/visual-token-pruning-in-multimodal-models/)

## Builds on (most cited first, 27 of 27)

- [MMBench: Is Your Multi-modal Model an All-around Player?](https://mlsanity.com/paper/2307.06281/) (2023)
- [MMMU: A Massive Multi-discipline Multimodal Understanding and Reasoning Benchmark for Expert AGI](https://mlsanity.com/paper/2311.16502/) (2023)
- [Learn to Explain: Multimodal Reasoning via Thought Chains for Science Question Answering](https://mlsanity.com/paper/2209.09513/) (2022)
- [MME: A Comprehensive Evaluation Benchmark for Multimodal Large Language Models](https://mlsanity.com/paper/2306.13394/) (2023)
- [SEED-Bench: Benchmarking Multimodal LLMs with Generative Comprehension](https://mlsanity.com/paper/2307.16125/) (2023)
- [VizWiz Grand Challenge: Answering Visual Questions from Blind People](https://mlsanity.com/paper/1802.08218/) (2018)
- [An Image is Worth 1/2 Tokens After Layer 2: Plug-and-Play Inference Acceleration for Large Vision-Language Models](https://mlsanity.com/paper/2403.06764/) (2024)
- [QuaRot: Outlier-Free 4-Bit Inference in Rotated LLMs](https://mlsanity.com/paper/2404.00456/) (2024)
- [VLMEvalKit: An Open-Source Toolkit for Evaluating Large Multi-Modality Models](https://mlsanity.com/paper/2407.11691/) (2024)
- [LLaVA-PruMerge: Adaptive Token Reduction for Efficient Large Multimodal Models](https://mlsanity.com/paper/2403.15388/) (2024)
- [VisionZip: Longer is Better but Not Necessary in Vision Language Models](https://mlsanity.com/paper/2412.04467/) (2024)
- [DivPrune: Diversity-based Visual Token Pruning for Large Multimodal Models](https://mlsanity.com/paper/2503.02175/) (2025)
- [Beyond Text-Visual Attention: Exploiting Visual Cues for Effective Token Pruning in VLMs](https://mlsanity.com/paper/2412.01818/) (2024)
- [FlatQuant: Flatness Matters for LLM Quantization](https://mlsanity.com/paper/2410.09426/) (2024)
- [Token Pruning in Multimodal Large Language Models: Are We Solving the Right Problem?](https://mlsanity.com/paper/2502.11501/) (2025)
- [On the Impact of Calibration Data in Post-training Quantization and Pruning](https://mlsanity.com/paper/2311.09755/) (2023)
- [Q-VLM: Post-training Quantization for Large Vision-Language Models](https://mlsanity.com/paper/2410.08119/) (2024)
- [MBQ: Modality-Balanced Quantization for Large Vision-Language Models](https://mlsanity.com/paper/2412.19509/) (2024)
- [Feather the Throttle: Revisiting Visual Token Pruning for Vision-Language Model Acceleration](https://mlsanity.com/paper/2412.13180/) (2024)
- [Enhancing Computation Efficiency in Large Language Models through Weight and Activation Quantization](https://mlsanity.com/paper/2311.05161/) (2023)
- [SQAP-VLA: A Synergistic Quantization-Aware Pruning Framework for High-Performance Vision-Language-Action Models](https://mlsanity.com/paper/2509.09090/) (2025)
- [Fine-Grained Post-Training Quantization for Large Vision Language Models with Quantization-Aware Integrated Gradients](https://mlsanity.com/paper/2603.17809/) (2026)
- [CRISP: Pre-LLM Yet Text-Driven Visual Token Pruning for Efficient LVLM Inference](https://mlsanity.com/paper/2607.16326/) (2026)
- [Rethinking Practical and Efficient Quantization Calibration for Vision-Language Models](https://mlsanity.com/paper/2602.07899/) (2026)
- [Spectral Heat Flow for Conservative Token Condensation in Vision-Language Models](https://mlsanity.com/paper/2607.10640/) (2026)
- [QAPruner: Quantization-Aware Vision Token Pruning for Multimodal Large Language Models](https://mlsanity.com/paper/2604.02816/) (2026)
- [Towards Joint Quantization and Token Pruning of Vision-Language Models](https://mlsanity.com/paper/2604.17320/) (2026)

## Abstract (arXiv)

Vision language models have achieved strong performance across a wide range of multimodal applications, yet their substantial computational and memory costs hinder efficient deployment. Visual token pruning and post-training quantization reduce inference overhead along two complementary dimensions, namely sequence length and numerical precision. Existing workflows typically optimize these techniques independently or apply them sequentially. Their distinct optimization objectives leave critical interactions unaddressed and constrain the achievable compression performance. We revisit these designs and present P4Q, a practical co-design framework that jointly optimizes visual token pruning and low-bit quantization for efficient VLM inference. First, P4Q introduces a quantization-aware visual token selection strategy before the LLM. It applies fake quantization to copies of the features produced by the projector and selects visual tokens using statistics computed from these fake-quantized features, thereby conditioning the selector's feature-based decisions on simulated low-bit perturbations. Second, P4Q introduces a pruning-aware quantization calibration strategy. It uses the same selection strategy as pruning to calibrate the quantized model on the retained-token distribution, thereby aligning the calibration process with the pruned execution path used during deployment. By coupling these two components, P4Q achieves substantial inference speedups while maintaining comparable task performance, resulting in a better efficiency-accuracy trade-off than independently optimized pipelines. For instance, on LLaVA-NeXT, P4Q achieves an average end-to-end inference speedup of 2.8x across eight distinct test sets, while retaining higher accuracy than prior compression and quantization methods.