# PathSelect: Sequential Token Selection for Whole Slide Pathology

arXiv: https://arxiv.org/abs/2607.23631 · HTML: https://mlsanity.com/paper/2607.23631/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Placed in Medical vision-language models, one of 16 directions in Multimodal LLMs / vision-language models.
- Builds on 42 of its 54 references that are in the map.
- First posted on arXiv 26 Jul 2026.

## Placed in

- Multimodal LLMs / vision-language models / [Medical vision-language models](https://mlsanity.com/mllm/medical-vision-language-models/)
- Efficient LLM inference and serving / [Visual token pruning in multimodal models](https://mlsanity.com/inference/visual-token-pruning-in-multimodal-models/)

## Builds on (most cited first, 30 of 42)

- [GPT-4o System Card](https://mlsanity.com/paper/2410.21276/) (2024)
- [Categorical Reparameterization with Gumbel-Softmax](https://mlsanity.com/paper/1611.01144/) (2016)
- [Estimating or Propagating Gradients Through Stochastic Neurons for Conditional Computation](https://mlsanity.com/paper/1308.3432/) (2013)
- [LLaVA-Med: Training a Large Language-and-Vision Assistant for Biomedicine in One Day](https://mlsanity.com/paper/2306.00890/) (2023)
- [Attention-based Deep Multiple Instance Learning](https://mlsanity.com/paper/1802.04712/) (2018)
- [Token Merging: Your ViT But Faster](https://mlsanity.com/paper/2210.09461/) (2022)
- [DynamicViT: Efficient Vision Transformers with Dynamic Token Sparsification](https://mlsanity.com/paper/2106.02034/) (2021)
- [Data Efficient and Weakly Supervised Computational Pathology on Whole Slide Images](https://mlsanity.com/paper/2004.09666/) (2020)
- [TransMIL: Transformer based Correlated Multiple Instance Learning for Whole Slide Image Classification](https://mlsanity.com/paper/2106.00908/) (2021)
- [Scaling Vision Transformers to Gigapixel Images via Hierarchical Self-Supervised Learning](https://mlsanity.com/paper/2206.02647/) (2022)
- [VisionZip: Longer is Better but Not Necessary in Vision Language Models](https://mlsanity.com/paper/2412.04467/) (2024)
- [Stochastic Beams and Where to Find Them: The Gumbel-Top-k Trick for Sampling Sequences Without Replacement](https://mlsanity.com/paper/1903.06059/) (2019)
- [Beyond Attention or Similarity: Maximizing Conditional Diversity for Token Pruning in MLLMs](https://mlsanity.com/paper/2506.10967/) (2025)
- [Quilt-LLaVA: Visual Instruction Tuning by Extracting Localized Narratives from Open-Source Histopathology Videos](https://mlsanity.com/paper/2312.04746/) (2023)
- [A graph-transformer for whole slide image classification](https://mlsanity.com/paper/2205.09671/) (2022)
- [Feature Re-Embedding: Towards Foundation Model-Level Performance in Computational Pathology](https://mlsanity.com/paper/2402.17228/) (2024)
- [Dynamic-LLaVA: Efficient Multimodal Large Language Models via Dynamic Vision-language Context Sparsification](https://mlsanity.com/paper/2412.00876/) (2024)
- [SlideChat: A Large Vision-Language Assistant for Whole-Slide Pathology Image Understanding](https://mlsanity.com/paper/2410.11761/) (2024)
- [Multi-modal Gated Mixture of Local-to-Global Experts for Dynamic Image Fusion](https://mlsanity.com/paper/2302.01392/) (2023)
- [Streaming convolutional neural networks for end-to-end learning with multi-megapixel images](https://mlsanity.com/paper/1911.04432/) (2019)
- [WSI-VQA: Interpreting Whole Slide Images by Generative Visual Question Answering](https://mlsanity.com/paper/2407.05603/) (2024)
- [CPath-Omni: A Unified Multimodal Foundation Model for Patch and Whole Slide Image Analysis in Computational Pathology](https://mlsanity.com/paper/2412.12077/) (2024)
- [MMTok: Multimodal Coverage Maximization for Efficient Inference of VLMs](https://mlsanity.com/paper/2508.18264/) (2025)
- [Segmentation and Vascular Vectorization for Coronary Artery by Geometry-based Cascaded Neural Network](https://mlsanity.com/paper/2305.04208/) (2023)
- [FOCUS: Knowledge-enhanced Adaptive Visual Compression for Few-shot Whole Slide Image Classification](https://mlsanity.com/paper/2411.14743/) (2024)
- [HiDrop: Hierarchical Vision Token Reduction in MLLMs via Late Injection, Concave Pyramid Pruning, and Early Exit](https://mlsanity.com/paper/2602.23699/) (2026)
- [MedViLaM: A multimodal large language model with advanced generalizability and explainability for medical data understanding and generation](https://mlsanity.com/paper/2409.19684/) (2024)
- [TC-SSA: Token Compression via Semantic Slot Aggregation for Gigapixel Pathology Reasoning](https://mlsanity.com/paper/2603.01143/) (2026)
- [One Leaf Reveals the Season: Occlusion-Based Contrastive Learning with Semantic-Aware Views for Efficient Visual Representation](https://mlsanity.com/paper/2411.09858/) (2024)
- [Fewer Tokens, Greater Scaling: Self-Adaptive Visual Bases for Efficient and Expansive Representation Learning](https://mlsanity.com/paper/2511.19515/) (2025)

## Built on by (most cited first, 2 of 2)

- [DiffPrune: differentiable information throttling for token pruning in vision-language models](https://mlsanity.com/paper/2608.01985/) (2026)
- [StepPrune: Adaptive Sequential Visual Token Selection across Multimodal Large Language Models](https://mlsanity.com/paper/2609.13804/) (2026)

## Abstract (arXiv)

Gigapixel Whole-Slide Images (WSIs) present a fundamental computational bottleneck for vision-language models (VLMs) due to extreme sequence lengths. Existing approaches predominantly rely on spatial sampling or training-free pruning, which risk diluting weak but informative signals, leading to the loss of critical diagnostic evidence due to the spatially diffuse nature of pathological cues. We reformulate WSI token pruning as a sequential selection process, enabling the model to autonomously learn an optimal routing strategy rather than relying on static heuristics. We herein propose a decoupled routing framework integrated as an active plugin into the fully pre-trained SlideChat base model, leaving both the slide encoder and large language model frozen. To provide continuous gradients for the non-differentiable pruning operation during training, we introduce PathSelect. PathSelect employs a variance-preserving noise gate to modulate each patch's information flow via a differentiable Soft Top-K operator, paired with a diagonal-attention Denoiser that recovers the perturbed representations without semantic leakage. At inference, the PathSelect module is entirely detached. Relying solely on the trained Scorer, a deterministic Hard Top-K operator executes adaptive, data-dependent trajectory termination, significantly accelerating downstream generative processing with exceptionally low sequential token selection latency. Driven by an empirical average of only 44.86 tokens under a maximum constraint of K = 128, our framework achieves 74.00% overall accuracy on SlideBench (TCGA), representing an approximate 36.6x spatial token reduction relative to the uncompressed baseline average while consistently outperforming sampling-based counterparts.