# Free Lunch in Pathology Foundation Model: Task-specific Model Adaptation with Concept-Guided Feature Enhancement

arXiv: https://arxiv.org/abs/2411.09894 · HTML: https://mlsanity.com/paper/2411.09894/

## Key facts

- Cited by 11 papers in the map (data through 14 Oct 2026).
- Builds on 28 of its 42 references that are in the map.
- First posted on arXiv 15 Nov 2024.

## Builds on (most cited first, 28 of 28)

- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [DINOv2: Learning Robust Visual Features without Supervision](https://mlsanity.com/paper/2304.07193/) (2023)
- [Representation Learning with Contrastive Predictive Coding](https://mlsanity.com/paper/1807.03748/) (2018)
- [Densely Connected Convolutional Networks](https://mlsanity.com/paper/1608.06993/) (2016)
- [Learning deep representations by mutual information estimation and maximization](https://mlsanity.com/paper/1808.06670/) (2018)
- [Attention-based Deep Multiple Instance Learning](https://mlsanity.com/paper/1802.04712/) (2018)
- [Deep Variational Information Bottleneck](https://mlsanity.com/paper/1612.00410/) (2016)
- [iBOT: Image BERT Pre-Training with Online Tokenizer](https://mlsanity.com/paper/2111.07832/) (2021)
- [Data Efficient and Weakly Supervised Computational Pathology on Whole Slide Images](https://mlsanity.com/paper/2004.09666/) (2020)
- [Self-Normalizing Neural Networks](https://mlsanity.com/paper/1706.02515/) (2017)
- [TransMIL: Transformer based Correlated Multiple Instance Learning for Whole Slide Image Classification](https://mlsanity.com/paper/2106.00908/) (2021)
- [Dual-stream Multiple Instance Learning Network for Whole Slide Image Classification with Self-supervised Contrastive Learning](https://mlsanity.com/paper/2011.08939/) (2020)
- [Scaling Vision Transformers to Gigapixel Images via Hierarchical Self-Supervised Learning](https://mlsanity.com/paper/2206.02647/) (2022)
- [DTFD-MIL: Double-Tier Feature Distillation Multiple Instance Learning for Histopathology Whole Slide Image Classification](https://mlsanity.com/paper/2203.12081/) (2022)
- [Quilt-1M: One Million Image-Text Pairs for Histopathology](https://mlsanity.com/paper/2306.11207/) (2023)
- [Learning Robust Representations via Multi-View Information Bottleneck](https://mlsanity.com/paper/2002.07017/) (2020)
- [Model-Based Domain Generalization](https://mlsanity.com/paper/2102.11436/) (2021)
- [Virchow: A Million-Slide Digital Pathology Foundation Model](https://mlsanity.com/paper/2309.07778/) (2023)
- [Feature Re-Embedding: Towards Foundation Model-Level Performance in Computational Pathology](https://mlsanity.com/paper/2402.17228/) (2024)
- [Additive MIL: Intrinsically Interpretable Multiple Instance Learning for Pathology](https://mlsanity.com/paper/2206.01794/) (2022)
- [Deep learning-based transformation of the H&E stain into special stains](https://mlsanity.com/paper/2008.08871/) (2020)
- [Fine-Tuning and Training of DenseNet for Histopathology Image Representation Using TCGA Diagnostic Slides](https://mlsanity.com/paper/2101.07903/) (2021)
- [Text-guided Foundation Model Adaptation for Pathological Image Classification](https://mlsanity.com/paper/2307.14901/) (2023)
- [The Rise of AI Language Pathologists: Exploring Two-level Prompt Learning for Few-shot Weakly-supervised Whole Slide Image Classification](https://mlsanity.com/paper/2305.17891/) (2023)
- [On Information Plane Analyses of Neural Network Classifiers -- A Review](https://mlsanity.com/paper/2003.09671/) (2020)
- [Generalizable Whole Slide Image Classification with Fine-Grained Visual-Semantic Interaction](https://mlsanity.com/paper/2402.19326/) (2024)
- [ConSlide: Asynchronous Hierarchical Interaction Transformer with Breakup-Reorganize Rehearsal for Continual Whole Slide Image Analysis](https://mlsanity.com/paper/2308.13324/) (2023)
- [PathoTune: Adapting Visual Foundation Model to Pathological Specialists](https://mlsanity.com/paper/2403.16497/) (2024)

## Built on by (most cited first, 11 of 11)

- [Multimodal Model for Computational Pathology:Representation Learning and Image Compression](https://mlsanity.com/paper/2603.18660/) (2026)
- [Fusion of Multi-scale Heterogeneous Pathology Foundation Models for Whole Slide Image Analysis](https://mlsanity.com/paper/2510.27237/) (2025)
- [RankByGene: Gene-Guided Histopathology Representation Learning Through Cross-Modal Ranking Consistency](https://mlsanity.com/paper/2411.15076/) (2024)
- [Lifelong Whole Slide Image Analysis: Online Vision-Language Adaptation and Past-to-Present Gradient Distillation](https://mlsanity.com/paper/2505.01984/) (2025)
- [HyperPath: Knowledge-Guided Hyperbolic Semantic Hierarchy Modeling for WSI Analysis](https://mlsanity.com/paper/2506.16398/) (2025)
- [MergeSlide: Continual Model Merging and Task-to-Class Prompt-Aligned Inference for Lifelong Learning on Whole Slide Images](https://mlsanity.com/paper/2511.13099/) (2025)
- [ZeroSlide: Is Zero-Shot Classification Adequate for Lifelong Learning in Whole-Slide Image Analysis in the Era of Pathology Vision-Language Foundation Models?](https://mlsanity.com/paper/2504.15627/) (2025)
- [Exploiting Low-Dimensional Manifold of Features for Few-Shot Whole Slide Image Classification](https://mlsanity.com/paper/2505.15504/) (2025)
- [PAST: A multimodal single-cell foundation model for histopathology and spatial transcriptomics in cancer](https://mlsanity.com/paper/2507.06418/) (2025)
- [CGRL: Concept-Guided Pruning and Representation Learning for Whole-Slide Image Classification](https://mlsanity.com/paper/2607.12556/) (2026)
- [Role-Guided MOE for Encoder-Level Pathology Representation Learning in WSI Classification](https://mlsanity.com/paper/2609.34897/) (2026)

## Abstract (arXiv)

Whole slide image (WSI) analysis is gaining prominence within the medical imaging field. Recent advances in pathology foundation models have shown the potential to extract powerful feature representations from WSIs for downstream tasks. However, these foundation models are usually designed for general-purpose pathology image analysis and may not be optimal for specific downstream tasks or cancer types. In this work, we present Concept Anchor-guided Task-specific Feature Enhancement (CATE), an adaptable paradigm that can boost the expressivity and discriminativeness of pathology foundation models for specific downstream tasks. Based on a set of task-specific concepts derived from the pathology vision-language model with expert-designed prompts, we introduce two interconnected modules to dynamically calibrate the generic image features extracted by foundation models for certain tasks or cancer types. Specifically, we design a Concept-guided Information Bottleneck module to enhance task-relevant characteristics by maximizing the mutual information between image features and concept anchors while suppressing superfluous information. Moreover, a Concept-Feature Interference module is proposed to utilize the similarity between calibrated features and concept anchors to further generate discriminative task-specific features. The extensive experiments on public WSI datasets demonstrate that CATE significantly enhances the performance and generalizability of MIL models. Additionally, heatmap and umap visualization results also reveal the effectiveness and interpretability of CATE. The source code is available at https://github.com/HKU-MedAI/CATE.