# Prompt-based Dynamic Token Pruning for Efficient Segmentation of Medical Images

arXiv: https://arxiv.org/abs/2506.16369 · HTML: https://mlsanity.com/paper/2506.16369/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 20 of its 36 references that are in the map.
- First posted on arXiv 19 Jun 2025.

## Builds on (most cited first, 20 of 20)

- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [U-Net: Convolutional Networks for Biomedical Image Segmentation](https://mlsanity.com/paper/1505.04597/) (2015)
- [Mask R-CNN](https://mlsanity.com/paper/1703.06870/) (2017)
- [Layer Normalization](https://mlsanity.com/paper/1607.06450/) (2016)
- [SegFormer: Simple and Efficient Design for Semantic Segmentation with Transformers](https://mlsanity.com/paper/2105.15203/) (2021)
- [Visual Prompt Tuning](https://mlsanity.com/paper/2203.12119/) (2022)
- [TransUNet: Transformers Make Strong Encoders for Medical Image Segmentation](https://mlsanity.com/paper/2102.04306/) (2021)
- [UNETR: Transformers for 3D Medical Image Segmentation](https://mlsanity.com/paper/2103.10504/) (2021)
- [DynamicViT: Efficient Vision Transformers with Dynamic Token Sparsification](https://mlsanity.com/paper/2106.02034/) (2021)
- [Swin UNETR: Swin Transformers for Semantic Segmentation of Brain Tumors in MRI Images](https://mlsanity.com/paper/2201.01266/) (2022)
- [Evo-ViT: Slow-Fast Token Evolution for Dynamic Vision Transformer](https://mlsanity.com/paper/2108.01390/) (2021)
- [Structured Pruning for Deep Convolutional Neural Networks: A survey](https://mlsanity.com/paper/2303.00566/) (2023)
- [Shunted Self-Attention via Multi-Scale Token Aggregation](https://mlsanity.com/paper/2111.15193/) (2021)
- [Combo Loss: Handling Input and Output Imbalance in Multi-Organ Segmentation](https://mlsanity.com/paper/1805.02798/) (2018)
- [Dynamic Token Pruning in Plain Vision Transformers for Semantic Segmentation](https://mlsanity.com/paper/2308.01045/) (2023)
- [Beyond Masking: Demystifying Token-Based Pre-Training for Vision Transformers](https://mlsanity.com/paper/2203.14313/) (2022)
- [The Lighter The Better: Rethinking Transformers in Medical Image Segmentation Through Adaptive Pruning](https://mlsanity.com/paper/2206.14413/) (2022)
- [Token Sparsification for Faster Medical Image Segmentation](https://mlsanity.com/paper/2303.06522/) (2023)
- [A General and Efficient Training for Transformer via Token Expansion](https://mlsanity.com/paper/2404.00672/) (2024)
- [Structured Model Pruning for Efficient Inference in Computational Pathology](https://mlsanity.com/paper/2404.08831/) (2024)

## Abstract (arXiv)

The high computational demands of Vision Transformers (ViTs) in processing a large number of tokens often constrain their practical application in analyzing medical images. This research proposes a Prompt-driven Adaptive Token (it PrATo) pruning method to selectively reduce the processing of irrelevant tokens in the segmentation pipeline. The prompt-based spatial prior helps to rank the tokens according to their relevance. Tokens with low-relevance scores are down-weighted, ensuring that only the relevant ones are propagated for processing across subsequent stages. This data-driven pruning strategy improves segmentation accuracy and inference speed by allocating computational resources to essential regions. The proposed framework is integrated with several state-of-the-art models to facilitate the elimination of irrelevant tokens, thereby enhancing computational efficiency while preserving segmentation accuracy. The experimental results show a reduction of ~ 35-55% tokens; thus reducing the computational costs relative to baselines. Cost-effective medical image processing, using our framework, facilitates real-time diagnosis by expanding its applicability in resource-constrained environments.