# When and What to Prune? Stage-Aware Visual Token Pruning for Efficient VLA

arXiv: https://arxiv.org/abs/2610.05273 · HTML: https://mlsanity.com/paper/2610.05273/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Placed in Visual token pruning and efficiency, one of 16 directions in Multimodal LLMs / vision-language models.
- Builds on 32 of its 36 references that are in the map.
- First posted on arXiv 4 Oct 2026.

## Placed in

- Multimodal LLMs / vision-language models / [Visual token pruning and efficiency](https://mlsanity.com/mllm/visual-token-pruning-and-efficiency/)
- Efficient LLM inference and serving / [Visual token pruning in multimodal models](https://mlsanity.com/inference/visual-token-pruning-in-multimodal-models/)

## Builds on (most cited first, 30 of 32)

- [RT-2: Vision-Language-Action Models Transfer Web Knowledge to Robotic Control](https://mlsanity.com/paper/2307.15818/) (2023)
- [OpenVLA: An Open-Source Vision-Language-Action Model](https://mlsanity.com/paper/2406.09246/) (2024)
- [π0: A Vision-Language-Action Flow Model for General Robot Control](https://mlsanity.com/paper/2410.24164/) (2024)
- [RT-1: Robotics Transformer for Real-World Control at Scale](https://mlsanity.com/paper/2212.06817/) (2022)
- [π0.5: a Vision-Language-Action Model with Open-World Generalization](https://mlsanity.com/paper/2504.16054/) (2025)
- [Octo: An Open-Source Generalist Robot Policy](https://mlsanity.com/paper/2405.12213/) (2024)
- [LIBERO: Benchmarking Knowledge Transfer for Lifelong Robot Learning](https://mlsanity.com/paper/2306.03310/) (2023)
- [Token Merging: Your ViT But Faster](https://mlsanity.com/paper/2210.09461/) (2022)
- [Fine-Tuning Vision-Language-Action Models: Optimizing Speed and Success](https://mlsanity.com/paper/2502.19645/) (2025)
- [DynamicViT: Efficient Vision Transformers with Dynamic Token Sparsification](https://mlsanity.com/paper/2106.02034/) (2021)
- [Open X-Embodiment: Robotic Learning Datasets and RT-X Models](https://mlsanity.com/paper/2310.08864/) (2023)
- [An Image is Worth 1/2 Tokens After Layer 2: Plug-and-Play Inference Acceleration for Large Vision-Language Models](https://mlsanity.com/paper/2403.06764/) (2024)
- [Evaluating Real-World Robot Manipulation Policies in Simulation](https://mlsanity.com/paper/2405.05941/) (2024)
- [SparseVLM: Visual Token Sparsification for Efficient Vision-Language Model Inference](https://mlsanity.com/paper/2410.04417/) (2024)
- [Not All Patches are What You Need: Expediting Vision Transformers via Token Reorganizations](https://mlsanity.com/paper/2202.07800/) (2022)
- [DivPrune: Diversity-based Visual Token Pruning for Large Multimodal Models](https://mlsanity.com/paper/2503.02175/) (2025)
- [Beyond Text-Visual Attention: Exploiting Visual Cues for Effective Token Pruning in VLMs](https://mlsanity.com/paper/2412.01818/) (2024)
- [Boosting Multimodal Large Language Models with Visual Tokens Withdrawal for Rapid Inference](https://mlsanity.com/paper/2405.05803/) (2024)
- [Fit and Prune: Fast and Training-free Visual Token Pruning for Multi-modal Large Language Models](https://mlsanity.com/paper/2409.10197/) (2024)
- [EfficientVLA: Training-Free Acceleration and Compression for Vision-Language-Action Models](https://mlsanity.com/paper/2506.10100/) (2025)
- [VLA-Cache: Efficient Vision-Language-Action Manipulation via Adaptive Token Caching](https://mlsanity.com/paper/2502.02175/) (2025)
- [SP-VLA: A Joint Model Scheduling and Token Pruning Approach for VLA Model Acceleration](https://mlsanity.com/paper/2506.12723/) (2025)
- [Don't Just Chase "Highlighted Tokens" in MLLMs: Revisiting Visual Holistic Context Retention](https://mlsanity.com/paper/2510.02912/) (2025)
- [SpecPrune-VLA: Accelerating Vision-Language-Action Models via Action-Aware Self-Speculative Pruning](https://mlsanity.com/paper/2509.05614/) (2025)
- [Action-aware Dynamic Pruning for Efficient Vision-Language-Action Manipulation](https://mlsanity.com/paper/2509.22093/) (2025)
- [LVPruning: An Effective yet Simple Language-Guided Vision Token Pruning Approach for Multi-modal Large Language Models](https://mlsanity.com/paper/2501.13652/) (2025)
- [VFlowOpt: A Token Pruning Framework for LMMs with Visual Information Flow-Guided Optimization](https://mlsanity.com/paper/2508.05211/) (2025)
- [Balanced Token Pruning: Accelerating Vision Language Models Beyond Local Optimization](https://mlsanity.com/paper/2505.22038/) (2025)
- [FoPru: Focal Pruning for Efficient Large Vision-Language Models](https://mlsanity.com/paper/2411.14164/) (2024)
- [TAMP: Token-Adaptive Layerwise Pruning in Multimodal Large Language Models](https://mlsanity.com/paper/2504.09897/) (2025)

## Abstract (arXiv)

Visual token pruning is an effective way to accelerate vision-language models and is especially useful for vision-language-action (VLA) inference, where many visual tokens must be processed before predicting robot actions. Existing pruning methods usually estimate which tokens can be pruned based on attention scores or feature diversity, retaining tokens that are either highly attended or visually different from others. However, most of them use fixed pruning schedules, such as pruning once at a preset layer or pruning at uniformly spaced layers. Such schedules can be risky for VLA models, because the model may not know which visual regions matter for the action in early layers. Tokens that look unimportant at first may become useful after the model combines visual observations with the language instruction. In this work, we propose SAPrune, a training-free visual token pruning framework for efficient VLA inference. Instead of pruning at fixed layers, SAPrune uses a small calibration set to observe how action-to-visual attention changes across layers, and chooses pruning layers only after the attention pattern becomes more reliable. At each selected layer, SAPrune applies a dual-path pruning rule: one path protects strongly attended visual tokens from pruning, while the other prevents useful surrounding context from being discarded. Experiments on LIBERO, SIMPLER, and real-world robotic tasks show that SAPrune prunes 87.5% of visual tokens and achieves up to 1.718x inference speedup while maintaining competitive task success rates.