# Prune2Drive: A Plug-and-Play Framework for Accelerating Vision-Language Models in Autonomous Driving

arXiv: https://arxiv.org/abs/2508.13305 · HTML: https://mlsanity.com/paper/2508.13305/

## Key facts

- Cited by 16 papers in the map (data through 14 Oct 2026).
- Placed in Driving VLMs and open-vocabulary detection, one of 16 directions in Multimodal LLMs / vision-language models.
- Builds on 32 of its 42 references that are in the map.
- First posted on arXiv 18 Aug 2025.

## Placed in

- Multimodal LLMs / vision-language models / [Driving VLMs and open-vocabulary detection](https://mlsanity.com/mllm/driving-vlms-and-open-vocabulary-detection/)
- End-to-end autonomous driving / [VLMs for autonomous driving](https://mlsanity.com/driving/vlms-for-autonomous-driving/)

## Builds on (most cited first, 30 of 32)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [nuScenes: A multimodal dataset for autonomous driving](https://mlsanity.com/paper/1903.11027/) (2019)
- [FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness](https://mlsanity.com/paper/2205.14135/) (2022)
- [LLaVA-OneVision: Easy Visual Task Transfer](https://mlsanity.com/paper/2408.03326/) (2024)
- [FlashAttention-2: Faster Attention with Better Parallelism and Work Partitioning](https://mlsanity.com/paper/2307.08691/) (2023)
- [InternVL: Scaling up Vision Foundation Models and Aligning for Generic Visual-Linguistic Tasks](https://mlsanity.com/paper/2312.14238/) (2023)
- [DynamicViT: Efficient Vision Transformers with Dynamic Token Sparsification](https://mlsanity.com/paper/2106.02034/) (2021)
- [A Survey of Quantization Methods for Efficient Neural Network Inference](https://mlsanity.com/paper/2103.13630/) (2021)
- [An Image is Worth 1/2 Tokens After Layer 2: Plug-and-Play Inference Acceleration for Large Vision-Language Models](https://mlsanity.com/paper/2403.06764/) (2024)
- [Be Your Own Teacher: Improve the Performance of Convolutional Neural Networks via Self Distillation](https://mlsanity.com/paper/1905.08094/) (2019)
- [SparseVLM: Visual Token Sparsification for Efficient Vision-Language Model Inference](https://mlsanity.com/paper/2410.04417/) (2024)
- [Kimi-VL Technical Report](https://mlsanity.com/paper/2504.07491/) (2025)
- [Not All Patches are What You Need: Expediting Vision Transformers via Token Reorganizations](https://mlsanity.com/paper/2202.07800/) (2022)
- [LMDrive: Closed-Loop End-to-End Driving with Large Language Models](https://mlsanity.com/paper/2312.07488/) (2023)
- [Textual Explanations for Self-Driving Vehicles](https://mlsanity.com/paper/1807.11546/) (2018)
- [An Introduction to Vision-Language Modeling](https://mlsanity.com/paper/2405.17247/) (2024)
- [TokenPacker: Efficient Visual Projector for Multimodal LLM](https://mlsanity.com/paper/2407.02392/) (2024)
- [Stop Looking for Important Tokens in Multimodal Language Models: Duplication Matters More](https://mlsanity.com/paper/2502.11494/) (2025)
- [OmniDrive: A Holistic Vision-Language Dataset for Autonomous Driving with Counterfactual Reasoning](https://mlsanity.com/paper/2504.04348/) (2025)
- [Learned Token Pruning for Transformers](https://mlsanity.com/paper/2107.00910/) (2021)
- [LLM4Drive: A Survey of Large Language Models for Autonomous Driving](https://mlsanity.com/paper/2311.01043/) (2023)
- [EfficientVLA: Training-Free Acceleration and Compression for Vision-Language-Action Models](https://mlsanity.com/paper/2506.10100/) (2025)
- [VLM-AD: End-to-End Autonomous Driving through Vision-Language Model Supervision](https://mlsanity.com/paper/2412.14446/) (2024)
- [Token Pruning in Multimodal Large Language Models: Are We Solving the Right Problem?](https://mlsanity.com/paper/2502.11501/) (2025)
- [Multi-Stage Vision Token Dropping: Towards Efficient Multimodal Large Language Model](https://mlsanity.com/paper/2411.10803/) (2024)
- [MJ-Bench: Is Your Multimodal Reward Model Really a Good Judge for Text-to-Image Generation?](https://mlsanity.com/paper/2407.04842/) (2024)
- [PACT: Pruning and Clustering-Based Token Reduction for Faster Visual Language Models](https://mlsanity.com/paper/2504.08966/) (2025)
- [Video Compression Commander: Plug-and-Play Inference Acceleration for Video Large Language Models](https://mlsanity.com/paper/2505.14454/) (2025)
- [The Devil behind the mask: An emergent safety vulnerability of Diffusion LLMs](https://mlsanity.com/paper/2507.11097/) (2025)

## Built on by (most cited first, 16 of 16)

- [Accelerating Streaming Video Large Language Models via Hierarchical Token Compression](https://mlsanity.com/paper/2512.00891/) (2025)
- [Variation-aware Vision Token Dropping for Faster Large Vision-Language Models](https://mlsanity.com/paper/2509.01552/) (2025)
- [Efficient Multi-modal Large Language Models via Progressive Consistency Distillation](https://mlsanity.com/paper/2510.00515/) (2025)
- [AudioMarathon: A Comprehensive Benchmark for Long-Context Audio Understanding and Efficiency in Audio LLMs](https://mlsanity.com/paper/2510.07293/) (2025)
- [AI for Service: Proactive Assistance with AI Glasses](https://mlsanity.com/paper/2510.14359/) (2025)
- [Are We Using the Right Benchmark: An Evaluation Framework for Visual Token Compression Methods](https://mlsanity.com/paper/2510.07143/) (2025)
- [Innovator-VL: A Multimodal Large Language Model for Scientific Discovery](https://mlsanity.com/paper/2601.19325/) (2026)
- [IPCV: Information-Preserving Compression for MLLM Visual Encoders](https://mlsanity.com/paper/2512.18747/) (2025)
- [EvoStreaming: Your Offline Video Model Is a Natively Streaming Assistant](https://mlsanity.com/paper/2605.10343/) (2026)
- [EarlyTom: Early Token Compression Completes Fast Video Understanding](https://mlsanity.com/paper/2605.30010/) (2026)
- [OmniDocLayout: Towards Diverse Document Layout Generation via Coarse-to-Fine LLM Learning](https://mlsanity.com/paper/2510.26213/) (2025)
- [ST-Prune: Training-Free Spatio-Temporal Token Pruning for Vision-Language Models in Autonomous Driving](https://mlsanity.com/paper/2604.19145/) (2026)
- [MVPruner: Dynamic Token Pruning for Accelerating Multi-view Vision-Language Models in Autonomous Driving](https://mlsanity.com/paper/2606.27660/) (2026)
- [Plug-and-Play Volumetric Reconstruction for Compressive Sensing Light-Sheet Microscopy](https://mlsanity.com/paper/2607.01654/) (2026)
- [SlackDrive: Reclaiming Runtime Slack for Adaptive Driving Inference](https://mlsanity.com/paper/2609.28064/) (2026)
- [GeoGR: Spatial-Semantic Token Compression and Group Routing for Efficient 3D VLMs](https://mlsanity.com/paper/2610.13035/) (2026)

## Abstract (arXiv)

Vision-Language Models (VLMs) have emerged as a promising paradigm in autonomous driving (AD), providing a unified framework for perception and decision-making. However, their real-world deployment is hindered by significant computational overhead when processing high-resolution, multi-view images. This complexity stems from the massive number of visual tokens, which increases inference latency and memory consumption due to the quadratic complexity of self-attention. To address these challenges, we propose Prune2Drive, a plug-and-play visual token pruning framework for multi-view VLMs in AD. Prune2Drive introduces two core innovations: (i) a diversity-aware token selection mechanism that prioritizes semantic and spatial coverage across views, and (ii) a view-adaptive pruning controller that automatically learns optimal pruning ratios based on camera importance to downstream tasks. Unlike prior methods, Prune2Drive requires no model retraining or access to attention maps, ensuring compatibility with modern efficient attention implementations. Extensive experiments on the DriveLM and DriveLMM-o1 benchmarks demonstrate that Prune2Drive achieves significant speedups and memory savings with minimal performance impact. When retaining only 10% of visual tokens, our method achieves a 6.40x speedup in the prefilling phase and consumes only 13.4% of the original FLOPs, with a mere 3% average performance drop on the DriveLM benchmark. Code is available at: https://github.com/MinhaoXiong/Prune2Drive.git