# ViLa-MIL: Dual-scale Vision-Language Multiple Instance Learning for Whole Slide Image Classification

arXiv: https://arxiv.org/abs/2502.08391 · HTML: https://mlsanity.com/paper/2502.08391/

## Key facts

- Cited by 28 papers in the map (data through 14 Oct 2026).
- Builds on 38 of its 50 references that are in the map.
- First posted on arXiv 12 Feb 2025.

## Builds on (most cited first, 30 of 38)

- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [GPT-4 Technical Report](https://mlsanity.com/paper/2303.08774/) (2023)
- [LLaMA: Open and Efficient Foundation Language Models](https://mlsanity.com/paper/2302.13971/) (2023)
- [PaLM: Scaling Language Modeling with Pathways](https://mlsanity.com/paper/2204.02311/) (2022)
- [BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generation](https://mlsanity.com/paper/2201.12086/) (2022)
- [Fast Graph Representation Learning with PyTorch Geometric](https://mlsanity.com/paper/1903.02428/) (2019)
- [Learning to Prompt for Vision-Language Models](https://mlsanity.com/paper/2109.01134/) (2021)
- [Attention-based Deep Multiple Instance Learning](https://mlsanity.com/paper/1802.04712/) (2018)
- [CLIP-Adapter: Better Vision-Language Models with Feature Adapters](https://mlsanity.com/paper/2110.04544/) (2021)
- [Data Efficient and Weakly Supervised Computational Pathology on Whole Slide Images](https://mlsanity.com/paper/2004.09666/) (2020)
- [MaPLe: Multi-modal Prompt Learning](https://mlsanity.com/paper/2210.03117/) (2022)
- [RegionCLIP: Region-based Language-Image Pretraining](https://mlsanity.com/paper/2112.09106/) (2021)
- [TransMIL: Transformer based Correlated Multiple Instance Learning for Whole Slide Image Classification](https://mlsanity.com/paper/2106.00908/) (2021)
- [DenseCLIP: Language-Guided Dense Prediction with Context-Aware Prompting](https://mlsanity.com/paper/2112.01518/) (2021)
- [Dual-stream Multiple Instance Learning Network for Whole Slide Image Classification with Self-supervised Contrastive Learning](https://mlsanity.com/paper/2011.08939/) (2020)
- [Scaling Language-Image Pre-training via Masking](https://mlsanity.com/paper/2212.00794/) (2022)
- [DTFD-MIL: Double-Tier Feature Distillation Multiple Instance Learning for Histopathology Whole Slide Image Classification](https://mlsanity.com/paper/2203.12081/) (2022)
- [Quilt-1M: One Million Image-Text Pairs for Histopathology](https://mlsanity.com/paper/2306.11207/) (2023)
- [Visual Language Pretrained Multiple Instance Zero-Shot Transfer for Histopathology Images](https://mlsanity.com/paper/2306.07831/) (2023)
- [CORA: Adapting CLIP for Open-Vocabulary Detection with Region Prompting and Anchor Pre-Matching](https://mlsanity.com/paper/2303.13076/) (2023)
- [Task Residual for Tuning Vision-Language Models](https://mlsanity.com/paper/2211.10277/) (2022)
- [CLIP is Also an Efficient Segmenter: A Text-Driven Approach for Weakly Supervised Semantic Segmentation](https://mlsanity.com/paper/2212.09506/) (2022)
- [Artificial Intelligence for Digital and Computational Pathology](https://mlsanity.com/paper/2401.06148/) (2023)
- [Interventional Bag Multi-Instance Learning On Whole-Slide Pathological Images](https://mlsanity.com/paper/2303.06873/) (2023)
- [A graph-transformer for whole slide image classification](https://mlsanity.com/paper/2205.09671/) (2022)
- [CAMEL: A Weakly Supervised Learning Framework for Histopathology Image Segmentation](https://mlsanity.com/paper/1908.10555/) (2019)
- [E^2VPT: An Effective and Efficient Approach for Visual Prompt Tuning](https://mlsanity.com/paper/2307.13770/) (2023)
- [Multiple Instance Learning Framework with Masked Hard Instance Mining for Whole Slide Image Classification](https://mlsanity.com/paper/2307.15254/) (2023)
- [Task-specific Fine-tuning via Variational Information Bottleneck for Weakly-supervised Pathology Whole Slide Image Classification](https://mlsanity.com/paper/2303.08446/) (2023)

## Built on by (most cited first, 28 of 28)

- [Multi-Resolution Pathology-Language Pre-training Model with Text-Guided Visual Representation](https://mlsanity.com/paper/2504.18856/) (2025)
- [Efficient Whole Slide Pathology VQA via Token Compression](https://mlsanity.com/paper/2507.14497/) (2025)
- [DPsurv: Dual-Prototype Evidential Fusion for Uncertainty-Aware and Interpretable Whole-Slide Image Survival Prediction](https://mlsanity.com/paper/2510.00053/) (2025)
- [CARE: A Molecular-Guided Foundation Model with Adaptive Region Modeling for Whole Slide Image Analysis](https://mlsanity.com/paper/2602.21637/) (2026)
- [PathVQ: Reforming Computational Pathology Foundation Model for Whole Slide Image Analysis via Vector Quantization](https://mlsanity.com/paper/2503.06482/) (2025)
- [Few-Shot Learning from Gigapixel Images via Hierarchical Vision-Language Alignment and Modeling](https://mlsanity.com/paper/2505.17982/) (2025)
- [PTCMIL: Multiple Instance Learning via Prompt Token Clustering for Whole Slide Image Analysis](https://mlsanity.com/paper/2507.18848/) (2025)
- [LPD: Learnable Prototypes with Diversity Regularization for Weakly Supervised Histopathology Segmentation](https://mlsanity.com/paper/2512.05922/) (2025)
- [Exploiting Low-Dimensional Manifold of Features for Few-Shot Whole Slide Image Classification](https://mlsanity.com/paper/2505.15504/) (2025)
- [GMAT: Grounded Multi-Agent Clinical Description Generation for Text Encoder in Vision-Language MIL for Whole Slide Image Classification](https://mlsanity.com/paper/2508.01293/) (2025)
- [Multiple Instance Learning Framework with Masked Hard Instance Mining for Gigapixel Histopathology Image Analysis](https://mlsanity.com/paper/2509.11526/) (2025)
- [MAPLE: Multi-scale Attribute-enhanced Prompt Learning for Few-shot Whole Slide Image Classification](https://mlsanity.com/paper/2509.25863/) (2025)
- [MUSE: Harnessing Precise and Diverse Semantics for Few-Shot Whole Slide Image Classification](https://mlsanity.com/paper/2602.20873/) (2026)
- [Identifying actionable driver mutations in lung cancer using an efficient Asymmetric Transformer Decoder](https://mlsanity.com/paper/2508.02431/) (2025)
- [MOC: Meta-Optimized Classifier for Few-Shot Whole Slide Image Classification](https://mlsanity.com/paper/2508.09967/) (2025)
- [Boosting Pathology Foundation Models via Few-shot Prompt-tuning for Rare Cancer Subtyping](https://mlsanity.com/paper/2508.15904/) (2025)
- [Revisiting Data Challenges of Computational Pathology: A Pack-based Multiple Instance Learning Training Framework](https://mlsanity.com/paper/2509.20923/) (2025)
- [DualProtoSeg: Simple and Efficient Design with Text- and Image-Guided Prototype Learning for Weakly Supervised Histopathology Image Segmentation](https://mlsanity.com/paper/2512.10314/) (2025)
- [WSD-MIL: Window Scale Decay Multiple Instance Learning for Whole Slide Image Classification](https://mlsanity.com/paper/2512.19982/) (2025)
- [Does medical specialization of VLMs enhance discriminative power?: A comprehensive investigation through feature distribution analysis](https://mlsanity.com/paper/2601.14774/) (2026)
- [AINet: Anchor Instances Learning for Regional Heterogeneity in Whole Slide Image](https://mlsanity.com/paper/2602.20187/) (2026)
- [Layout-Guided Controllable Pathology Image Generation with In-Context Diffusion Transformers](https://mlsanity.com/paper/2603.13386/) (2026)
- [Parameter-efficient Prompt Tuning and Hierarchical Textual Guidance for Few-shot Whole Slide Image Classification](https://mlsanity.com/paper/2603.21504/) (2026)
- [Spatial Blindness in Whole-Slide Multiple Instance Learning](https://mlsanity.com/paper/2605.17449/) (2026)
- [TaxoMIL: Taxonomy-Constrained Learning for Hierarchical Whole Slide Image Analysis](https://mlsanity.com/paper/2606.31100/) (2026)
- [From Patches to Evidence Balls: Class-Conditioned Evidence Retrieval for Few-Shot Whole Slide Image Classification](https://mlsanity.com/paper/2608.01104/) (2026)
- [From Multi-Resolution Cells to Gigapixel Whole Slide Images Foundation Model for Computational Pathology](https://mlsanity.com/paper/2608.03508/) (2026)
- [FFM-CP: Cross-Backbone Fusion of Vision-Language Foundation Models for Few-Shot Computational Pathology](https://mlsanity.com/paper/2609.27710/) (2026)

## Abstract (arXiv)

Multiple instance learning (MIL)-based framework has become the mainstream for processing the whole slide image (WSI) with giga-pixel size and hierarchical image context in digital pathology. However, these methods heavily depend on a substantial number of bag-level labels and solely learn from the original slides, which are easily affected by variations in data distribution. Recently, vision language model (VLM)-based methods introduced the language prior by pre-training on large-scale pathological image-text pairs. However, the previous text prompt lacks the consideration of pathological prior knowledge, therefore does not substantially boost the model's performance. Moreover, the collection of such pairs and the pre-training process are very time-consuming and source-intensive.To solve the above problems, we propose a dual-scale vision-language multiple instance learning (ViLa-MIL) framework for whole slide image classification. Specifically, we propose a dual-scale visual descriptive text prompt based on the frozen large language model (LLM) to boost the performance of VLM effectively. To transfer the VLM to process WSI efficiently, for the image branch, we propose a prototype-guided patch decoder to aggregate the patch features progressively by grouping similar patches into the same prototype; for the text branch, we introduce a context-guided text decoder to enhance the text features by incorporating the multi-granular image contexts. Extensive studies on three multi-cancer and multi-center subtyping datasets demonstrate the superiority of ViLa-MIL.