# Prompt-Guided Foundation Model Tuning for Pathology Image Classification

arXiv: https://arxiv.org/abs/2403.12537 · HTML: https://mlsanity.com/paper/2403.12537/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- Builds on 33 of its 66 references that are in the map.
- First posted on arXiv 19 Mar 2024.

## Builds on (most cited first, 30 of 33)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [LoRA: Low-Rank Adaptation of Large Language Models](https://mlsanity.com/paper/2106.09685/) (2021)
- [A Simple Framework for Contrastive Learning of Visual Representations](https://mlsanity.com/paper/2002.05709/) (2020)
- [Visual Prompt Tuning](https://mlsanity.com/paper/2203.12119/) (2022)
- [Attention-based Deep Multiple Instance Learning](https://mlsanity.com/paper/1802.04712/) (2018)
- [Few-Shot Parameter-Efficient Fine-Tuning is Better and Cheaper than In-Context Learning](https://mlsanity.com/paper/2205.05638/) (2022)
- [AdaptFormer: Adapting Vision Transformers for Scalable Visual Recognition](https://mlsanity.com/paper/2205.13535/) (2022)
- [TransMIL: Transformer based Correlated Multiple Instance Learning for Whole Slide Image Classification](https://mlsanity.com/paper/2106.00908/) (2021)
- [Image Segmentation Using Text and Image Prompts](https://mlsanity.com/paper/2112.10003/) (2021)
- [Dual-stream Multiple Instance Learning Network for Whole Slide Image Classification with Self-supervised Contrastive Learning](https://mlsanity.com/paper/2011.08939/) (2020)
- [Rotation Equivariant CNNs for Digital Pathology](https://mlsanity.com/paper/1806.03962/) (2018)
- [Exploring Visual Prompts for Adapting Large-Scale Models](https://mlsanity.com/paper/2203.17274/) (2022)
- [Deep neural network models for computational histopathology: A survey](https://mlsanity.com/paper/1912.12378/) (2019)
- [Patch-based Convolutional Neural Network for Whole Slide Tissue Image Classification](https://mlsanity.com/paper/1504.07947/) (2015)
- [Visual Language Pretrained Multiple Instance Zero-Shot Transfer for Histopathology Images](https://mlsanity.com/paper/2306.07831/) (2023)
- [BACH: Grand Challenge on Breast Cancer Histology Images](https://mlsanity.com/paper/1808.04277/) (2018)
- [MambaMIL: Enhancing Long Sequence Modeling with Sequence Reordering in Computational Pathology](https://mlsanity.com/paper/2403.06800/) (2024)
- [Multi-scale Domain-adversarial Multiple-instance CNN for Cancer Subtype Classification with Unannotated Histopathological Images](https://mlsanity.com/paper/2001.01599/) (2020)
- [MELO: Enhancing Model Editing with Neuron-Indexed Dynamic LoRA](https://mlsanity.com/paper/2312.11795/) (2023)
- [Dynamic Graph Representation with Knowledge-aware Attention for Histopathology Whole Slide Image Analysis](https://mlsanity.com/paper/2403.07719/) (2024)
- [Predicting breast tumor proliferation from whole-slide images: the TUPAC16 challenge](https://mlsanity.com/paper/1807.08284/) (2018)
- [Hard Sample Aware Noise Robust Learning for Histopathology Image Classification](https://mlsanity.com/paper/2112.03694/) (2021)
- [Weakly supervised multiple instance learning histopathological tumor segmentation](https://mlsanity.com/paper/2004.05024/) (2020)
- [UniToPatho, a labeled histopathological dataset for colorectal polyps classification and adenoma dysplasia grading](https://mlsanity.com/paper/2101.09991/) (2021)
- [ReMix: A General and Efficient Framework for Multiple Instance Learning based Whole Slide Image Classification](https://mlsanity.com/paper/2207.01805/) (2022)
- [ConSlide: Asynchronous Hierarchical Interaction Transformer with Breakup-Reorganize Rehearsal for Continual Whole Slide Image Analysis](https://mlsanity.com/paper/2308.13324/) (2023)

## Built on by (most cited first, 4 of 4)

- [Cohort-Individual Cooperative Learning for Multimodal Cancer Survival Analysis](https://mlsanity.com/paper/2404.02394/) (2024)
- [Efficient Whole Slide Image Classification through Fisher Vector Representation](https://mlsanity.com/paper/2411.08530/) (2024)
- [Prompt Mechanisms in Medical Imaging: A Comprehensive Survey](https://mlsanity.com/paper/2507.01055/) (2025)
- [TAP-Path: Task-Adaptive Structural and Token Pruning for Efficient and Trustworthy Pathology Foundation Models](https://mlsanity.com/paper/2609.04071/) (2026)

## Abstract (arXiv)

Foundation models have become pivotal in advancing computational pathology, particularly for whole slide image (WSI) classification. However, prevailing methodologies often rely on frozen, pre-trained models for feature extraction, overlooking the pronounced domain shift and task discrepancy between the pre-training and downstream tasks. To address this challenge, we propose PAMT, a novel Prompt-guided Adaptive Model Transformation framework that enables precise adaptation of general foundation models to the distinct domain of histopathology. To encapsulate the intricate distributions characteristic of histopathological data, we introduce Representative Patch Sampling (RPS) and Prototypical Visual Prompt (PVP), which reconstruct the input into compact yet highly informative representations. Further, to effectively bridge the domain gap, we incorporate Adaptive Model Transformation (AMT) via adapter modules within the feature extraction pipeline, facilitating the acquisition of domain-specific features by the foundation model. We conduct rigorous evaluation across 14 publicly available datasets and demonstrate consistent, substantial improvements in classification accuracy. These results establish PAMT as a compelling new benchmark for pathology image classification and underscore the critical value of targeted model adaptation within computational pathology.