# Beyond Surrogate Gradients: Fully Differentiable Token Pruning for Vision-Language Models

arXiv: https://arxiv.org/abs/2605.28051 · HTML: https://mlsanity.com/paper/2605.28051/

## Key facts

- Cited by 7 papers in the map (data through 14 Oct 2026).
- Placed in Visual token pruning and efficiency, one of 16 directions in Multimodal LLMs / vision-language models.
- Builds on 20 of its 76 references that are in the map.
- First posted on arXiv 27 May 2026.

## Placed in

- Multimodal LLMs / vision-language models / [Visual token pruning and efficiency](https://mlsanity.com/mllm/visual-token-pruning-and-efficiency/)
- Efficient LLM inference and serving / [Visual token pruning in multimodal models](https://mlsanity.com/inference/visual-token-pruning-in-multimodal-models/)

## Builds on (most cited first, 20 of 20)

- [Qwen2.5-VL Technical Report](https://mlsanity.com/paper/2502.13923/) (2025)
- [The Better You Learn, The Smarter You Prune: Towards Efficient Vision-language-action Models via Differentiable Token Pruning](https://mlsanity.com/paper/2509.12594/) (2025)
- [Similarity-Aware Token Pruning: Your VLM but Faster](https://mlsanity.com/paper/2503.11549/) (2025)
- [A Glimpse to Compress: Dynamic Visual Token Pruning for Large Vision-Language Models](https://mlsanity.com/paper/2508.01548/) (2025)
- [MedViLaM: A multimodal large language model with advanced generalizability and explainability for medical data understanding and generation](https://mlsanity.com/paper/2409.19684/) (2024)
- [TC-SSA: Token Compression via Semantic Slot Aggregation for Gigapixel Pathology Reasoning](https://mlsanity.com/paper/2603.01143/) (2026)
- [Fewer Tokens, Greater Scaling: Self-Adaptive Visual Bases for Efficient and Expansive Representation Learning](https://mlsanity.com/paper/2511.19515/) (2025)
- [Growing a Multi-head Twig via Distillation and Reinforcement Learning to Accelerate Large Vision-Language Models](https://mlsanity.com/paper/2503.14075/) (2025)
- [Learning A Multi-Task Transformer Via Unified And Customized Instruction Tuning For Chest Radiograph Interpretation](https://mlsanity.com/paper/2311.01092/) (2023)
- [A foundation model for generalizable disease diagnosis in chest X-ray images](https://mlsanity.com/paper/2410.08861/) (2024)
- [The Model Knows Which Tokens Matter: Automatic Token Selection via Noise Gating](https://mlsanity.com/paper/2603.07135/) (2026)
- [ZeroSense:How Vision matters in Long Context Compression](https://mlsanity.com/paper/2603.11846/) (2026)
- [Dynamic Token Reduction during Generation for Vision Language Models](https://mlsanity.com/paper/2501.14204/) (2025)
- [Efficient Chest X-ray Representation Learning via Semantic-Partitioned Contrastive Learning](https://mlsanity.com/paper/2603.07113/) (2026)
- [Fine-Grained VLM Fine-tuning via Latent Hierarchical Adapter Learning](https://mlsanity.com/paper/2508.11176/) (2025)
- [Multimodal Model for Computational Pathology:Representation Learning and Image Compression](https://mlsanity.com/paper/2603.18660/) (2026)
- [Improving Discrete Optimisation Via Decoupled Straight-Through Estimator](https://mlsanity.com/paper/2410.13331/) (2024)
- [Bi-VLM: Pushing Ultra-Low Precision Post-Training Quantization Boundaries in Vision-Language Models](https://mlsanity.com/paper/2509.18763/) (2025)
- [LKV: End-to-End Learning of Head-wise Budgets and Token Selection for LLM KV Cache Eviction](https://mlsanity.com/paper/2605.06676/) (2026)
- [Shiva-DiT: Residual-Based Differentiable Top-k Selection for Efficient Diffusion Transformers](https://mlsanity.com/paper/2602.05605/) (2026)

## Built on by (most cited first, 7 of 7)

- [Stepwise Token Selection for Efficient Multimodal Large Language Models](https://mlsanity.com/paper/2606.16067/) (2026)
- [Learnable Token Sparsification for Efficient Gigapixel Whole Slide Image Reasoning](https://mlsanity.com/paper/2606.08641/) (2026)
- [PathSelect: Sequential Token Selection for Whole Slide Pathology](https://mlsanity.com/paper/2607.23631/) (2026)
- [Decoupling semantics from vision: A framework for faithful visual-text compression evaluation](https://mlsanity.com/paper/2608.01848/) (2026)
- [DiffPrune: differentiable information throttling for token pruning in vision-language models](https://mlsanity.com/paper/2608.01985/) (2026)
- [StepPrune: Adaptive Sequential Visual Token Selection across Multimodal Large Language Models](https://mlsanity.com/paper/2609.13804/) (2026)
- [VLA-ACL: Action-Consistent Visual Token Pruning for Efficient Vision-Language-Action Models](https://mlsanity.com/paper/2610.08133/) (2026)

## Abstract (arXiv)

Visual token pruning reduces the computational cost of Vision-Language Models (VLMs) by removing redundant visual tokens. Existing methods typically rely on Gumbel-Softmax to approximate discrete selection during training. However, the optimization is driven by surrogate gradients rather than the true selection process, leading to unreliable learning of token importance. In this paper, we propose DiffPrune, which reformulates pruning as continuous control of token information instead of discrete selection learning. Specifically, we introduce an Information Throttler that modulates each token using variance-preserving noise conditioned on importance scores, where higher scores induce less information suppression during training. This design directly operates on token representations, naturally providing a fully differentiable optimization path for learning token importance. At inference, tokens are removed via hard thresholding on the learned scores. Across ten VLM benchmarks, DiffPrune retains 96.5% of full-model accuracy while accelerating LLM prefill by 2.85x, with only 0.69 ms of inference overhead.