# Few-Shot Learning from Gigapixel Images via Hierarchical Vision-Language Alignment and Modeling

arXiv: https://arxiv.org/abs/2505.17982 · HTML: https://mlsanity.com/paper/2505.17982/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Placed in Medical vision-language models, one of 16 directions in Multimodal LLMs / vision-language models.
- Builds on 37 of its 58 references that are in the map.
- First posted on arXiv 23 May 2025.

## Placed in

- Multimodal LLMs / vision-language models / [Medical vision-language models](https://mlsanity.com/mllm/medical-vision-language-models/)

## Builds on (most cited first, 30 of 37)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [PyTorch: An Imperative Style, High-Performance Deep Learning Library](https://mlsanity.com/paper/1912.01703/) (2019)
- [GPT-4 Technical Report](https://mlsanity.com/paper/2303.08774/) (2023)
- [Inductive Representation Learning on Large Graphs](https://mlsanity.com/paper/1706.02216/) (2017)
- [Flamingo: a Visual Language Model for Few-Shot Learning](https://mlsanity.com/paper/2204.14198/) (2022)
- [BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generation](https://mlsanity.com/paper/2201.12086/) (2022)
- [Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervision](https://mlsanity.com/paper/2102.05918/) (2021)
- [Fast Graph Representation Learning with PyTorch Geometric](https://mlsanity.com/paper/1903.02428/) (2019)
- [Learning to Prompt for Vision-Language Models](https://mlsanity.com/paper/2109.01134/) (2021)
- [Graph Neural Networks: A Review of Methods and Applications](https://mlsanity.com/paper/1812.08434/) (2018)
- [A Survey on LLM-as-a-Judge](https://mlsanity.com/paper/2411.15594/) (2024)
- [Relational inductive biases, deep learning, and graph networks](https://mlsanity.com/paper/1806.01261/) (2018)
- [Attention-based Deep Multiple Instance Learning](https://mlsanity.com/paper/1802.04712/) (2018)
- [CLIP-Adapter: Better Vision-Language Models with Feature Adapters](https://mlsanity.com/paper/2110.04544/) (2021)
- [Data Efficient and Weakly Supervised Computational Pathology on Whole Slide Images](https://mlsanity.com/paper/2004.09666/) (2020)
- [Contrastive Learning of Medical Visual Representations from Paired Images and Text](https://mlsanity.com/paper/2010.00747/) (2020)
- [TransMIL: Transformer based Correlated Multiple Instance Learning for Whole Slide Image Classification](https://mlsanity.com/paper/2106.00908/) (2021)
- [Dual-stream Multiple Instance Learning Network for Whole Slide Image Classification with Self-supervised Contrastive Learning](https://mlsanity.com/paper/2011.08939/) (2020)
- [Rethinking Few-Shot Image Classification: a Good Embedding Is All You Need?](https://mlsanity.com/paper/2003.11539/) (2020)
- [A Baseline for Few-Shot Image Classification](https://mlsanity.com/paper/1909.02729/) (2019)
- [DTFD-MIL: Double-Tier Feature Distillation Multiple Instance Learning for Histopathology Whole Slide Image Classification](https://mlsanity.com/paper/2203.12081/) (2022)
- [Quilt-1M: One Million Image-Text Pairs for Histopathology](https://mlsanity.com/paper/2306.11207/) (2023)
- [Multiple Instance Learning: A Survey of Problem Characteristics and Applications](https://mlsanity.com/paper/1612.03365/) (2016)
- [Interventional Bag Multi-Instance Learning On Whole-Slide Pathological Images](https://mlsanity.com/paper/2303.06873/) (2023)
- [A graph-transformer for whole slide image classification](https://mlsanity.com/paper/2205.09671/) (2022)
- [Dynamic Graph Representation with Knowledge-aware Attention for Histopathology Whole Slide Image Analysis](https://mlsanity.com/paper/2403.07719/) (2024)
- [Multiple Instance Learning for Digital Pathology: A Review on the State-of-the-Art, Limitations & Future Potential](https://mlsanity.com/paper/2206.04425/) (2022)
- [The Rise of AI Language Pathologists: Exploring Two-level Prompt Learning for Few-shot Weakly-supervised Whole Slide Image Classification](https://mlsanity.com/paper/2305.17891/) (2023)
- [DGMIL: Distribution Guided Multiple Instance Learning for Whole Slide Image Classification](https://mlsanity.com/paper/2206.08861/) (2022)

## Built on by (most cited first, 3 of 3)

- [Beyond Relevance: Bayesian Evidence Acquisition for Agentic Whole-Slide Image Reasoning](https://mlsanity.com/paper/2608.05757/) (2026)
- [Spatial Blindness in Whole-Slide Multiple Instance Learning](https://mlsanity.com/paper/2605.17449/) (2026)
- [From Patches to Evidence Balls: Class-Conditioned Evidence Retrieval for Few-Shot Whole Slide Image Classification](https://mlsanity.com/paper/2608.01104/) (2026)

## Abstract (arXiv)

Vision-language models (VLMs) have recently been integrated into multiple instance learning (MIL) frameworks to address the challenge of few-shot, weakly supervised classification of whole slide images (WSIs). A key trend involves leveraging multi-scale information to better represent hierarchical tissue structures. However, existing methods often face two key limitations: (1) insufficient modeling of interactions within the same modalities across scales (e.g., 5x and 20x) and (2) inadequate alignment between visual and textual modalities on the same scale. To address these gaps, we propose HiVE-MIL, a hierarchical vision-language framework that constructs a unified graph consisting of (1) parent-child links between coarse (5x) and fine (20x) visual/textual nodes to capture hierarchical relationships, and (2) heterogeneous intra-scale edges linking visual and textual nodes on the same scale. To further enhance semantic consistency, HiVE-MIL incorporates a two-stage, text-guided dynamic filtering mechanism that removes weakly correlated patch-text pairs, and introduces a hierarchical contrastive loss to align textual semantics across scales. Extensive experiments on TCGA breast, lung, and kidney cancer datasets demonstrate that HiVE-MIL consistently outperforms both traditional MIL and recent VLM-based MIL approaches, achieving gains of up to 4.1% in macro F1 under 16-shot settings. Our results demonstrate the value of jointly modeling hierarchical structure and multimodal alignment for efficient and scalable learning from limited pathology data. The code is available at https://github.com/bryanwong17/HiVE-MIL.