# The Better You Learn, The Smarter You Prune: Towards Efficient Vision-language-action Models via Differentiable Token Pruning

arXiv: https://arxiv.org/abs/2509.12594 · HTML: https://mlsanity.com/paper/2509.12594/

## Key facts

- Cited by 36 papers in the map (data through 14 Oct 2026).
- Placed in Efficient VLA inference and adaptation, one of 13 directions in Vision-language-action models.
- Builds on 42 of its 45 references that are in the map.
- First posted on arXiv 16 Sep 2025.

## Placed in

- Vision-language-action models / [Efficient VLA inference and adaptation](https://mlsanity.com/vla/efficient-vla-inference-and-adaptation/)
- Efficient LLM inference and serving / [Visual token pruning in multimodal models](https://mlsanity.com/inference/visual-token-pruning-in-multimodal-models/)

## Builds on (most cited first, 30 of 42)

- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [LoRA: Low-Rank Adaptation of Large Language Models](https://mlsanity.com/paper/2106.09685/) (2021)
- [LLaMA: Open and Efficient Foundation Language Models](https://mlsanity.com/paper/2302.13971/) (2023)
- [DINOv2: Learning Robust Visual Features without Supervision](https://mlsanity.com/paper/2304.07193/) (2023)
- [BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Models](https://mlsanity.com/paper/2301.12597/) (2023)
- [Efficient Memory Management for Large Language Model Serving with PagedAttention](https://mlsanity.com/paper/2309.06180/) (2023)
- [Improved Baselines with Visual Instruction Tuning](https://mlsanity.com/paper/2310.03744/) (2023)
- [Qwen2-VL: Enhancing Vision-Language Model's Perception of the World at Any Resolution](https://mlsanity.com/paper/2409.12191/) (2024)
- [Sigmoid Loss for Language Image Pre-Training](https://mlsanity.com/paper/2303.15343/) (2023)
- [OpenVLA: An Open-Source Vision-Language-Action Model](https://mlsanity.com/paper/2406.09246/) (2024)
- [InternVL: Scaling up Vision Foundation Models and Aligning for Generic Visual-Linguistic Tasks](https://mlsanity.com/paper/2312.14238/) (2023)
- [π0: A Vision-Language-Action Flow Model for General Robot Control](https://mlsanity.com/paper/2410.24164/) (2024)
- [π0.5: a Vision-Language-Action Model with Open-World Generalization](https://mlsanity.com/paper/2504.16054/) (2025)
- [Root Mean Square Layer Normalization](https://mlsanity.com/paper/1910.07467/) (2019)
- [LIBERO: Benchmarking Knowledge Transfer for Lifelong Robot Learning](https://mlsanity.com/paper/2306.03310/) (2023)
- [SGLang: Efficient Execution of Structured Language Model Programs](https://mlsanity.com/paper/2312.07104/) (2023)
- [Token Merging: Your ViT But Faster](https://mlsanity.com/paper/2210.09461/) (2022)
- [Fine-Tuning Vision-Language-Action Models: Optimizing Speed and Success](https://mlsanity.com/paper/2502.19645/) (2025)
- [DeepSeekMoE: Towards Ultimate Expert Specialization in Mixture-of-Experts Language Models](https://mlsanity.com/paper/2401.06066/) (2024)
- [FAST: Efficient Action Tokenization for Vision-Language-Action Models](https://mlsanity.com/paper/2501.09747/) (2025)
- [An Image is Worth 1/2 Tokens After Layer 2: Plug-and-Play Inference Acceleration for Large Vision-Language Models](https://mlsanity.com/paper/2403.06764/) (2024)
- [SmolVLA: A Vision-Language-Action Model for Affordable and Efficient Robotics](https://mlsanity.com/paper/2506.01844/) (2025)
- [CogACT: A Foundational Vision-Language-Action Model for Synergizing Cognition and Action in Robotic Manipulation](https://mlsanity.com/paper/2411.19650/) (2024)
- [LLaVA-PruMerge: Adaptive Token Reduction for Efficient Large Multimodal Models](https://mlsanity.com/paper/2403.15388/) (2024)
- [SparseVLM: Visual Token Sparsification for Efficient Vision-Language Model Inference](https://mlsanity.com/paper/2410.04417/) (2024)
- [WorldVLA: Towards Autoregressive Action World Model](https://mlsanity.com/paper/2506.21539/) (2025)
- [TinyVLA: Towards Fast, Data-Efficient Vision-Language-Action Models for Robotic Manipulation](https://mlsanity.com/paper/2409.12514/) (2024)
- [PyramidDrop: Accelerating Your Large Vision-Language Models via Pyramid Visual Redundancy Reduction](https://mlsanity.com/paper/2410.17247/) (2024)
- [MolmoAct: Action Reasoning Models that can Reason in Space](https://mlsanity.com/paper/2508.07917/) (2025)
- [LLaVA-Mini: Efficient Image and Video Large Multimodal Models with One Vision Token](https://mlsanity.com/paper/2501.03895/) (2025)

## Built on by (most cited first, 30 of 36)

- [VLASH: Real-Time VLAs via Future-State-Aware Asynchronous Inference](https://mlsanity.com/paper/2512.01031/) (2025)
- [A Survey on Efficient Vision-Language-Action Models](https://mlsanity.com/paper/2510.24795/) (2025)
- [Efficient Vision-Language-Action Models for Embodied Manipulation: A Systematic Survey](https://mlsanity.com/paper/2510.17111/) (2025)
- [AVA-VLA: Improving Vision-Language-Action models with Active Visual Attention](https://mlsanity.com/paper/2511.18960/) (2025)
- [FASTER: Rethinking Real-Time Flow VLAs](https://mlsanity.com/paper/2603.19199/) (2026)
- [DyQ-VLA: Temporal-Dynamic-Aware Quantization for Embodied Vision-Language-Action Models](https://mlsanity.com/paper/2603.07904/) (2026)
- [VisionSelector: End-to-End Learnable Visual Token Compression for Efficient Multimodal LLMs](https://mlsanity.com/paper/2510.16598/) (2025)
- [TurboVLA: Real-Time Vision-Language-Action Model at 32 Hz on an RTX 4090 with <1 GB VRAM](https://mlsanity.com/paper/2607.27205/) (2026)
- [Fast-dVLA: Accelerating Discrete Diffusion VLA to Real-Time Performance](https://mlsanity.com/paper/2603.25661/) (2026)
- [Shallow-π: Knowledge Distillation for Flow-based VLAs](https://mlsanity.com/paper/2601.20262/) (2026)
- [Token Expand-Merge: Training-Free Token Compression for Vision-Language-Action Models](https://mlsanity.com/paper/2512.09927/) (2025)
- [Beyond Surrogate Gradients: Fully Differentiable Token Pruning for Vision-Language Models](https://mlsanity.com/paper/2605.28051/) (2026)
- [AC^2-VLA: Action-Context-Aware Adaptive Computation in Vision-Language-Action Models for Efficient Robotic Manipulation](https://mlsanity.com/paper/2601.19634/) (2026)
- [FocusVLA: Focused Visual Utilization for Vision-Language-Action Models](https://mlsanity.com/paper/2603.28740/) (2026)
- [ActDistill: General Action-Guided Self-Derived Distillation for Efficient Vision-Language-Action Models](https://mlsanity.com/paper/2511.18082/) (2025)
- [OxyGen: Unified KV Cache Management for VLA Inference under Multi-Task Parallelism](https://mlsanity.com/paper/2603.14371/) (2026)
- [Latent Bridge: Feature Delta Prediction for Efficient Dual-System Vision-Language-Action Model Inference](https://mlsanity.com/paper/2605.02739/) (2026)
- [FlashVLA: Streaming Action Decoding for Fast and Asynchronous VLA Inference](https://mlsanity.com/paper/2608.27384/) (2026)
- [Jetson-PI: Towards Onboard Real-Time Robot Control via Foresight-Aligned Asynchronous Inference](https://mlsanity.com/paper/2607.12659/) (2026)
- [Compression Tells Intelligence: Visual Coding, Visual Token Technology, and the Unification](https://mlsanity.com/paper/2601.20742/) (2026)
- [From Inference Efficiency to Embodied Efficiency: Revisiting Efficiency Metrics for Vision-Language-Action Models](https://mlsanity.com/paper/2603.19131/) (2026)
- [ElegantVLA: Learning When to Think for Efficient Vision-Language-Action Models](https://mlsanity.com/paper/2605.29438/) (2026)
- [PolicyTrim: Boosting Intrinsic Policy Efficiency of Vision-Language-Action Models](https://mlsanity.com/paper/2606.22540/) (2026)
- [Training-Free Interaction-Aligned Visual Token Pruning for Efficient Embodied Manipulation](https://mlsanity.com/paper/2603.22991/) (2026)
- [UniFS: Unified Fast-to-Slow Hierarchical Architecture for Vision-Language-Action Models](https://mlsanity.com/paper/2606.22794/) (2026)
- [DiffPrune: differentiable information throttling for token pruning in vision-language models](https://mlsanity.com/paper/2608.01985/) (2026)
- [Think Proprioceptively: State-Grounded Visual Token Selection for VLA Policies](https://mlsanity.com/paper/2602.06575/) (2026)
- [VLA-InfoEntropy: A Training-Free Vision-Attention Information Entropy Approach for Vision-Language-Action Models Inference Acceleration and Success](https://mlsanity.com/paper/2604.05323/) (2026)
- [2D or 3D: Who Governs Salience in VLA Models? -- Tri-Stage Token Pruning Framework with Modality Salience Awareness](https://mlsanity.com/paper/2604.09244/) (2026)
- [LiteVLA-H: Dual-Rate Vision-Language-Action Inference for Onboard Aerial Guidance and Semantic Perception](https://mlsanity.com/paper/2605.00884/) (2026)

## Abstract (arXiv)

We present LightVLA, a simple yet effective differentiable token pruning framework for vision-language-action (VLA) models. While VLA models have shown impressive capability in executing real-world robotic tasks, their deployment on resource-constrained platforms is often bottlenecked by the heavy attention-based computation over large sets of visual tokens. LightVLA addresses this challenge through adaptive, performance-driven pruning of visual tokens: It generates dynamic queries to evaluate visual token importance, and adopts Gumbel softmax to enable differentiable token selection. Through fine-tuning, LightVLA learns to preserve the most informative visual tokens while pruning tokens which do not contribute to task execution, thereby improving efficiency and performance simultaneously. Notably, LightVLA requires no heuristic magic numbers and introduces no additional trainable parameters, making it compatible with modern inference frameworks. Experimental results demonstrate that LightVLA outperforms different VLA models and existing token pruning methods across diverse tasks on the LIBERO benchmark, achieving higher success rates with substantially reduced computational overhead. Specifically, LightVLA reduces FLOPs and latency by 59.1% and 38.2% respectively, with a 2.6% improvement in task success rate. Meanwhile, we also investigate the learnable query-based token pruning method LightVLA* with additional trainable parameters, which also achieves satisfactory performance. Our work reveals that as VLA pursues optimal performance, LightVLA spontaneously learns to prune tokens from a performance-driven perspective. To the best of our knowledge, LightVLA is the first work to apply adaptive visual token pruning to VLA tasks with the collateral goals of efficiency and performance, marking a significant step toward more efficient, powerful and practical real-time robotic systems.