# TAP-Path: Task-Adaptive Structural and Token Pruning for Efficient and Trustworthy Pathology Foundation Models

arXiv: https://arxiv.org/abs/2609.04071 · HTML: https://mlsanity.com/paper/2609.04071/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Placed in Visual token pruning in multimodal models, one of 12 directions in Efficient LLM inference and serving.
- Builds on 15 of its 33 references that are in the map.
- First posted on arXiv 3 Sep 2026.

## Placed in

- Efficient LLM inference and serving / [Visual token pruning in multimodal models](https://mlsanity.com/inference/visual-token-pruning-in-multimodal-models/)

## Builds on (most cited first, 15 of 15)

- [On Calibration of Modern Neural Networks](https://mlsanity.com/paper/1706.04599/) (2017)
- [Can You Trust Your Model's Uncertainty? Evaluating Predictive Uncertainty Under Dataset Shift](https://mlsanity.com/paper/1906.02530/) (2019)
- [SelectiveNet: A Deep Neural Network with an Integrated Reject Option](https://mlsanity.com/paper/1901.09192/) (2019)
- [A Clinical Benchmark of Public Self-Supervised Pathology Foundation Models](https://mlsanity.com/paper/2407.06508/) (2024)
- [Hibou: A Family of Foundational Vision Transformers for Pathology](https://mlsanity.com/paper/2406.05074/) (2024)
- [Revisiting Token Pruning for Object Detection and Instance Segmentation](https://mlsanity.com/paper/2306.07050/) (2023)
- [Benchmarking foundation models as feature extractors for weakly-supervised computational pathology](https://mlsanity.com/paper/2408.15823/) (2024)
- [Domain Generalization in Computational Pathology: Survey and Guidelines](https://mlsanity.com/paper/2310.19656/) (2023)
- [GTP-ViT: Efficient Vision Transformers via Graph-based Token Propagation](https://mlsanity.com/paper/2311.03035/) (2023)
- [Token Cropr: Faster ViTs for Quite a Few Tasks](https://mlsanity.com/paper/2412.00965/) (2024)
- [Towards Foundation Models and Few-Shot Parameter-Efficient Fine-Tuning for Volumetric Organ Segmentation](https://mlsanity.com/paper/2303.17051/) (2023)
- [Benchmarking Pathology Foundation Models: Adaptation Strategies and Scenarios](https://mlsanity.com/paper/2410.16038/) (2024)
- [PathOrchestra: A Comprehensive Foundation Model for Computational Pathology with Over 100 Diverse Clinical-Grade Tasks](https://mlsanity.com/paper/2503.24345/) (2025)
- [Prompt-Guided Foundation Model Tuning for Pathology Image Classification](https://mlsanity.com/paper/2403.12537/) (2024)
- [AdaptViG: Adaptive Vision GNN with Exponential Decay Gating](https://mlsanity.com/paper/2511.09942/) (2025)

## Abstract (arXiv)

Pathology foundation models improve transferable representation learning for histopathology, but recent gains often rely on encoders with hundreds of millions of parameters and high inference cost. We propose TAP-Path, a task-adaptive compression framework that directly restructures a pretrained Virchow2 encoder rather than distilling it into a separate student. TAP-Path combines validation-driven transformer-block selection, physical removal of redundant blocks, input-adaptive patch-token pruning, multi-depth feature recovery, and a lightweight gated task head. The final model retains 24 of 32 transformer blocks and 70% of patch tokens after pruning, reducing encoder parameters by 24.96% (631.24M to 473.70M) and analytical encoder compute by 35.20% (340.13G to 220.40G FLOPs). Across three task-head optimization seeds, TAP-Path achieved 87.98 ± 0.067% test accuracy, 81.26 ± 0.49% balanced accuracy, and 82.38 ± 0.48% macro-F1 on a 32-class histopathology benchmark, compared with 86.89% for full Virchow2 and 87.67% for UNI2-h. TAP-Path achieved a Brier score of 0.1800 ± 0.0005 and failure-detection AUROC of 0.9047 ± 0.0060. A validation-only rare-aware objective improved rare-class balanced accuracy in a secondary operating analysis. Frozen external evaluation on 433 CPTAC samples yielded 91.22 ± 0.83% accuracy and 91.10 ± 0.81% balanced accuracy. These results show that task-adaptive structural and token sparsification can improve the accuracy-efficiency trade-off of large pathology foundation models while preserving reliability under internal and external evaluation.