# PathoTune: Adapting Visual Foundation Model to Pathological Specialists

arXiv: https://arxiv.org/abs/2403.16497 · HTML: https://mlsanity.com/paper/2403.16497/

## Key facts

- Cited by 8 papers in the map (data through 14 Oct 2026).
- Builds on 22 of its 35 references that are in the map.
- First posted on arXiv 25 Mar 2024.

## Builds on (most cited first, 22 of 22)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [Momentum Contrast for Unsupervised Visual Representation Learning](https://mlsanity.com/paper/1911.05722/) (2019)
- [Masked Autoencoders Are Scalable Vision Learners](https://mlsanity.com/paper/2111.06377/) (2021)
- [Visual Prompt Tuning](https://mlsanity.com/paper/2203.12119/) (2022)
- [BitFit: Simple Parameter-efficient Fine-tuning for Transformer-based Masked Language-models](https://mlsanity.com/paper/2106.10199/) (2021)
- [On the Variance of the Adaptive Learning Rate and Beyond](https://mlsanity.com/paper/1908.03265/) (2019)
- [AdaptFormer: Adapting Vision Transformers for Scalable Visual Recognition](https://mlsanity.com/paper/2205.13535/) (2022)
- [Scaling Vision Transformers to Gigapixel Images via Hierarchical Self-Supervised Learning](https://mlsanity.com/paper/2206.02647/) (2022)
- [Self supervised contrastive learning for digital histopathology](https://mlsanity.com/paper/2011.13971/) (2020)
- [Virchow: A Million-Slide Digital Pathology Foundation Model](https://mlsanity.com/paper/2309.07778/) (2023)
- [Visual Prompt Tuning for Generative Transfer Learning](https://mlsanity.com/paper/2210.00990/) (2022)
- [Going Deeper through the Gleason Scoring Scale: An Automatic end-to-end System for Histology Prostate Grading and Cribriform Pattern Detection](https://mlsanity.com/paper/2105.10490/) (2021)
- [BCI: Breast Cancer Immunohistochemical Image Generation through Pyramid Pix2pix](https://mlsanity.com/paper/2204.11425/) (2022)
- [Visual Query Tuning: Towards Effective Usage of Intermediate Representations for Parameter and Memory Efficient Transfer Learning](https://mlsanity.com/paper/2212.03220/) (2022)
- [Text-guided Foundation Model Adaptation for Pathological Image Classification](https://mlsanity.com/paper/2307.14901/) (2023)
- [A General-Purpose Self-Supervised Model for Computational Pathology](https://mlsanity.com/paper/2308.15474/) (2023)
- [Classification and Disease Localization in Histopathology Using Only Global Labels: A Weakly-Supervised Approach](https://mlsanity.com/paper/1802.02212/) (2018)
- [PathoDuet: Foundation Models for Pathological Slide Analysis of H&E and IHC Stains](https://mlsanity.com/paper/2312.09894/) (2023)
- [Multiple Instance Learning with Mixed Supervision in Gleason Grading](https://mlsanity.com/paper/2206.12798/) (2022)
- [Bag Graph: Multiple Instance Learning using Bayesian Graph Neural Networks](https://mlsanity.com/paper/2202.11132/) (2022)

## Built on by (most cited first, 8 of 8)

- [Free Lunch in Pathology Foundation Model: Task-specific Model Adaptation with Concept-Guided Feature Enhancement](https://mlsanity.com/paper/2411.09894/) (2024)
- [A New Era in Computational Pathology: A Survey on Foundation and Vision-Language Models](https://mlsanity.com/paper/2408.14496/) (2024)
- [HuggingGraph: Understanding the Supply Chain of LLM Ecosystem](https://mlsanity.com/paper/2507.14240/) (2025)
- [Event USKT : U-State Space Model in Knowledge Transfer for Event Cameras](https://mlsanity.com/paper/2411.15276/) (2024)
- [CLIP-IT: CLIP-based Pairing for Histology Images Classification](https://mlsanity.com/paper/2504.16181/) (2025)
- [Hypergraph Mamba for Efficient Whole Slide Image Understanding](https://mlsanity.com/paper/2505.17457/) (2025)
- [Multi-Modal Parameter-Efficient Fine-tuning via Graph Neural Network](https://mlsanity.com/paper/2408.00290/) (2024)
- [Role-Guided MOE for Encoder-Level Pathology Representation Learning in WSI Classification](https://mlsanity.com/paper/2609.34897/) (2026)

## Abstract (arXiv)

As natural image understanding moves towards the pretrain-finetune era, research in pathology imaging is concurrently evolving. Despite the predominant focus on pretraining pathological foundation models, how to adapt foundation models to downstream tasks is little explored. For downstream adaptation, we propose the existence of two domain gaps, i.e., the Foundation-Task Gap and the Task-Instance Gap. To mitigate these gaps, we introduce PathoTune, a framework designed to efficiently adapt pathological or even visual foundation models to pathology-specific tasks via multi-modal prompt tuning. The proposed framework leverages Task-specific Visual Prompts and Task-specific Textual Prompts to identify task-relevant features, along with Instance-specific Visual Prompts for encoding single pathological image features. Results across multiple datasets at both patch-level and WSI-level demonstrate its superior performance over single-modality prompt tuning approaches. Significantly, PathoTune facilitates the direct adaptation of natural visual foundation models to pathological tasks, drastically outperforming pathological foundation models with simple linear probing. The code is available at https://github.com/openmedlab/PathoDuet.