# Text-guided Foundation Model Adaptation for Pathological Image Classification

arXiv: https://arxiv.org/abs/2307.14901 · HTML: https://mlsanity.com/paper/2307.14901/

## Key facts

- Cited by 47 papers in the map (data through 14 Oct 2026).
- Builds on 20 of its 31 references that are in the map.
- First posted on arXiv 27 Jul 2023.

## Builds on (most cited first, 20 of 20)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [ImageNet Large Scale Visual Recognition Challenge](https://mlsanity.com/paper/1409.0575/) (2014)
- [BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Models](https://mlsanity.com/paper/2301.12597/) (2023)
- [On the Opportunities and Risks of Foundation Models](https://mlsanity.com/paper/2108.07258/) (2021)
- [Learning to Prompt for Vision-Language Models](https://mlsanity.com/paper/2109.01134/) (2021)
- [BioBERT: a pre-trained biomedical language representation model for biomedical text mining](https://mlsanity.com/paper/1901.08746/) (2019)
- [Visual Prompt Tuning](https://mlsanity.com/paper/2203.12119/) (2022)
- [Publicly Available Clinical BERT Embeddings](https://mlsanity.com/paper/1904.03323/) (2019)
- [Data Efficient and Weakly Supervised Computational Pathology on Whole Slide Images](https://mlsanity.com/paper/2004.09666/) (2020)
- [LiT: Zero-Shot Transfer with Locked-image text Tuning](https://mlsanity.com/paper/2111.07991/) (2021)
- [LinkBERT: Pretraining Language Models with Document Links](https://mlsanity.com/paper/2203.15827/) (2022)
- [VisualGPT: Data-efficient Adaptation of Pretrained Language Models for Image Captioning](https://mlsanity.com/paper/2102.10407/) (2021)
- [Meta-Baseline: Exploring Simple Meta-Learning for Few-Shot Learning](https://mlsanity.com/paper/2003.04390/) (2020)
- [Learning Customized Visual Models with Retrieval-Augmented Knowledge](https://mlsanity.com/paper/2301.07094/) (2023)
- [INTERN: A New Learning Paradigm Towards General Vision](https://mlsanity.com/paper/2111.08687/) (2021)
- [Inference of captions from histopathological patches](https://mlsanity.com/paper/2202.03432/) (2022)

## Built on by (most cited first, 30 of 47)

- [A Comprehensive Survey of Scientific Large Language Models and Their Applications in Scientific Discovery](https://mlsanity.com/paper/2406.10833/) (2024)
- [Foundation Model for Advancing Healthcare: Challenges, Opportunities, and Future Directions](https://mlsanity.com/paper/2404.03264/) (2024)
- [Foundational Models in Medical Imaging: A Comprehensive Survey and Future Vision](https://mlsanity.com/paper/2310.18689/) (2023)
- [A Comprehensive Survey of Foundation Models in Medicine](https://mlsanity.com/paper/2406.10729/) (2024)
- [CLIP in Medical Imaging: A Survey](https://mlsanity.com/paper/2312.07353/) (2023)
- [PathAlign: A vision-language model for whole slide images in histopathology](https://mlsanity.com/paper/2406.19578/) (2024)
- [ViLa-MIL: Dual-scale Vision-Language Multiple Instance Learning for Whole Slide Image Classification](https://mlsanity.com/paper/2502.08391/) (2025)
- [Data-Centric Foundation Models in Computational Healthcare: A Survey](https://mlsanity.com/paper/2401.02458/) (2024)
- [Generalizable Whole Slide Image Classification with Fine-Grained Visual-Semantic Interaction](https://mlsanity.com/paper/2402.19326/) (2024)
- [Medical Vision Language Pretraining: A survey](https://mlsanity.com/paper/2312.06224/) (2023)
- [Benchmarking Pathology Foundation Models: Adaptation Strategies and Scenarios](https://mlsanity.com/paper/2410.16038/) (2024)
- [BAPLe: Backdoor Attacks on Medical Foundational Models using Prompt Learning](https://mlsanity.com/paper/2408.07440/) (2024)
- [PathoDuet: Foundation Models for Pathological Slide Analysis of H&E and IHC Stains](https://mlsanity.com/paper/2312.09894/) (2023)
- [PathM3: A Multimodal Multi-Task Multiple Instance Learning Framework for Whole Slide Image Classification and Captioning](https://mlsanity.com/paper/2403.08967/) (2024)
- [Pathology-knowledge Enhanced Multi-instance Prompt Learning for Few-shot Whole Slide Image Classification](https://mlsanity.com/paper/2407.10814/) (2024)
- [Free Lunch in Pathology Foundation Model: Task-specific Model Adaptation with Concept-Guided Feature Enhancement](https://mlsanity.com/paper/2411.09894/) (2024)
- [DeViDe: Faceted medical knowledge for improved medical vision-language pre-training](https://mlsanity.com/paper/2404.03618/) (2024)
- [A New Era in Computational Pathology: A Survey on Foundation and Vision-Language Models](https://mlsanity.com/paper/2408.14496/) (2024)
- [Taming Vision-Language Models for Medical Image Analysis: A Comprehensive Review](https://mlsanity.com/paper/2506.18378/) (2025)
- [PathoTune: Adapting Visual Foundation Model to Pathological Specialists](https://mlsanity.com/paper/2403.16497/) (2024)
- [FAST: A Dual-tier Few-Shot Learning Paradigm for Whole Slide Image Classification](https://mlsanity.com/paper/2409.19720/) (2024)
- [Foundation Models in Medical Imaging: A Review and Outlook](https://mlsanity.com/paper/2506.09095/) (2025)
- [Embedded Visual Prompt Tuning](https://mlsanity.com/paper/2407.01003/) (2024)
- [Cost-effective Instruction Learning for Pathology Vision and Language Analysis](https://mlsanity.com/paper/2407.17734/) (2024)
- [Clinical-grade Multi-Organ Pathology Report Generation for Multi-scale Whole Slide Images via a Semantically Guided Medical Text Foundation Model](https://mlsanity.com/paper/2409.15574/) (2024)
- [Multimodal Model for Computational Pathology:Representation Learning and Image Compression](https://mlsanity.com/paper/2603.18660/) (2026)
- [OpenMEDLab: An Open-source Platform for Multi-modality Foundation Models in Medicine](https://mlsanity.com/paper/2402.18028/) (2024)
- [Advancing COVID-19 Detection in 3D CT Scans](https://mlsanity.com/paper/2403.11953/) (2024)
- [CognoSpeak: an automatic, remote assessment of early cognitive decline in real-world conversational speech](https://mlsanity.com/paper/2501.05755/) (2025)
- [Diagnostic Text-guided Representation Learning in Hierarchical Classification for Pathological Whole Slide Image](https://mlsanity.com/paper/2411.10709/) (2024)

## Abstract (arXiv)

The recent surge of foundation models in computer vision and natural language processing opens up perspectives in utilizing multi-modal clinical data to train large models with strong generalizability. Yet pathological image datasets often lack biomedical text annotation and enrichment. Guiding data-efficient image diagnosis from the use of biomedical text knowledge becomes a substantial interest. In this paper, we propose to Connect Image and Text Embeddings (CITE) to enhance pathological image classification. CITE injects text insights gained from language models pre-trained with a broad range of biomedical texts, leading to adapt foundation models towards pathological image understanding. Through extensive experiments on the PatchGastric stomach tumor pathological image dataset, we demonstrate that CITE achieves leading performance compared with various baselines especially when training data is scarce. CITE offers insights into leveraging in-domain text knowledge to reinforce data-efficient pathological image classification. Code is available at https://github.com/Yunkun-Zhang/CITE.