mlsanityData to 7 Oct 2026

TAP-Path: Task-Adaptive Structural and Token Pruning for Efficient and Trustworthy Pathology Foundation Models

arXiv 2609.04071 ↗First posted 3 Sep 2026cs.CV cs.AIcited by 0

On the map: Efficient LLM inference and serving → Visual token pruning in multimodal models

Abstract (arXiv)

Pathology foundation models improve transferable representation learning for histopathology, but recent gains often rely on encoders with hundreds of millions of parameters and high inference cost. We propose TAP-Path, a task-adaptive compression framework that directly restructures a pretrained Virchow2 encoder rather than distilling it into a separate student. TAP-Path combines validation-driven transformer-block selection, physical removal of redundant blocks, input-adaptive patch-token pruning, multi-depth feature recovery, and a lightweight gated task head. The final model retains 24 of 32 transformer blocks and 70% of patch tokens after pruning, reducing encoder parameters by 24.96% (631.24M to 473.70M) and analytical encoder compute by 35.20% (340.13G to 220.40G FLOPs). Across three task-head optimization seeds, TAP-Path achieved 87.98 ± 0.067% test accuracy, 81.26 ± 0.49% balanced accuracy, and 82.38 ± 0.48% macro-F1 on a 32-class histopathology benchmark, compared with 86.89% for full Virchow2 and 87.67% for UNI2-h. TAP-Path achieved a Brier score of 0.1800 ± 0.0005 and failure-detection AUROC of 0.9047 ± 0.0060. A validation-only rare-aware objective improved rare-class balanced accuracy in a secondary operating analysis. Frozen external evaluation on 433 CPTAC samples yielded 91.22 ± 0.83% accuracy and 91.10 ± 0.81% balanced accuracy. These results show that task-adaptive structural and token sparsification can improve the accuracy-efficiency trade-off of large pathology foundation models while preserving reliability under internal and external evaluation.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Placed in Visual token pruning in multimodal models, one of 12 directions in Efficient LLM inference and serving. Builds on 15 of its 33 references that are in the map. First posted on arXiv 3 Sep 2026.

Builds on

15 of 33 references.

References · 15

  1. 2017On Calibration of Modern Neural Networkscited by 4,860
  2. 2019Can You Trust Your Model's Uncertainty? Evaluating Predictive Uncertainty Under Dataset Shiftcited by 1,274
  3. 2019SelectiveNet: A Deep Neural Network with an Integrated Reject Optioncited by 333
  4. 2024A Clinical Benchmark of Public Self-Supervised Pathology Foundation Modelscited by 64
  5. 2024Hibou: A Family of Foundational Vision Transformers for Pathologycited by 52
  6. 2023Revisiting Token Pruning for Object Detection and Instance Segmentationcited by 50
  7. 2024Benchmarking foundation models as feature extractors for weakly-supervised computational pathologycited by 47
  8. 2023Domain Generalization in Computational Pathology: Survey and Guidelinescited by 29
  9. 2023GTP-ViT: Efficient Vision Transformers via Graph-based Token Propagationcited by 19
  10. 2024Token Cropr: Faster ViTs for Quite a Few Taskscited by 18
  11. 2023Towards Foundation Models and Few-Shot Parameter-Efficient Fine-Tuning for Volumetric Organ Segmentationcited by 17
  12. 2024Benchmarking Pathology Foundation Models: Adaptation Strategies and Scenarioscited by 16
3 more
  1. 2025PathOrchestra: A Comprehensive Foundation Model for Computational Pathology with Over 100 Diverse Clinical-Grade Taskscited by 8
  2. 2024Prompt-Guided Foundation Model Tuning for Pathology Image Classificationcited by 4
  3. 2025AdaptViG: Adaptive Vision GNN with Exponential Decay Gatingcited by 2

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2025Pathology Image Compression with Pre-trained Autoencoders
  2. 2025Teaching pathology foundation models to accurately predict gene expression with parameter efficient knowledge transfer
  3. 2025LoC-Path: Learning to Compress for Pathology Multimodal Large Language Models
  4. 2025Prototype-Guided Diffusion for Digital Pathology: Achieving Foundation Model Performance with Minimal Clinical Data
  5. 2025PathDiff: Histopathology Image Synthesis with Unpaired Text and Mask Conditions
  6. 2026SLICEChat: Progressive In-Encoder Token Pruning for Whole-Slide Pathology Language Models
  7. 2024Boosting Vision-Language Models for Histopathology Classification: Predict all at once
  8. 2026Multimodal LLMs Outperform Pathology Foundation Models in Cross-Domain Histological Similarity