# In-context learning enables multimodal large language models to classify cancer pathology images

arXiv: https://arxiv.org/abs/2403.07407 · HTML: https://mlsanity.com/paper/2403.07407/

## Key facts

- Cited by 38 papers in the map (data through 14 Oct 2026).
- Placed in Medical vision-language models, one of 16 directions in Multimodal LLMs / vision-language models.
- Builds on 20 of its 46 references that are in the map.
- First posted on arXiv 12 Mar 2024.

## Placed in

- Multimodal LLMs / vision-language models / [Medical vision-language models](https://mlsanity.com/mllm/medical-vision-language-models/)

## Builds on (most cited first, 20 of 20)

- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [Chain-of-Thought Prompting Elicits Reasoning in Large Language Models](https://mlsanity.com/paper/2201.11903/) (2022)
- [Visual Instruction Tuning](https://mlsanity.com/paper/2304.08485/) (2023)
- [Grad-CAM: Visual Explanations from Deep Networks via Gradient-based Localization](https://mlsanity.com/paper/1610.02391/) (2016)
- [Flamingo: a Visual Language Model for Few-Shot Learning](https://mlsanity.com/paper/2204.14198/) (2022)
- [Tree of Thoughts: Deliberate Problem Solving with Large Language Models](https://mlsanity.com/paper/2305.10601/) (2023)
- [Training Compute-Optimal Large Language Models](https://mlsanity.com/paper/2203.15556/) (2022)
- [ZeRO: Memory Optimizations Toward Training Trillion Parameter Models](https://mlsanity.com/paper/1910.02054/) (2019)
- [Large Language Models as Optimizers](https://mlsanity.com/paper/2309.03409/) (2023)
- [Data Efficient and Weakly Supervised Computational Pathology on Whole Slide Images](https://mlsanity.com/paper/2004.09666/) (2020)
- [The Dawn of LMMs: Preliminary Explorations with GPT-4V(ision)](https://mlsanity.com/paper/2309.17421/) (2023)
- [Promptbreeder: Self-Referential Self-Improvement Via Prompt Evolution](https://mlsanity.com/paper/2309.16797/) (2023)
- [Scaling Laws for Autoregressive Generative Modeling](https://mlsanity.com/paper/2010.14701/) (2020)
- [Towards Expert-Level Medical Question Answering with Large Language Models](https://mlsanity.com/paper/2305.09617/) (2023)
- [Can Generalist Foundation Models Outcompete Special-Purpose Tuning? Case Study in Medicine](https://mlsanity.com/paper/2311.16452/) (2023)
- [Benchmarking Self-Supervised Learning on Diverse Pathology Datasets](https://mlsanity.com/paper/2212.04690/) (2022)
- [Virchow: A Million-Slide Digital Pathology Foundation Model](https://mlsanity.com/paper/2309.07778/) (2023)
- [A Petri Dish for Histopathology Image Analysis](https://mlsanity.com/paper/2101.12355/) (2021)
- [A General-Purpose Self-Supervised Model for Computational Pathology](https://mlsanity.com/paper/2308.15474/) (2023)
- [From Whole-slide Image to Biomarker Prediction: A Protocol for End-to-End Deep Learning in Computational Pathology](https://mlsanity.com/paper/2312.10944/) (2023)

## Built on by (most cited first, 30 of 38)

- [EndoChat: Grounded Multimodal Large Language Model for Endoscopic Surgery](https://mlsanity.com/paper/2501.11347/) (2025)
- [True Multimodal In-Context Learning Needs Attention to the Visual Context](https://mlsanity.com/paper/2507.15807/) (2025)
- [A New Era in Computational Pathology: A Survey on Foundation and Vision-Language Models](https://mlsanity.com/paper/2408.14496/) (2024)
- [Taming Vision-Language Models for Medical Image Analysis: A Comprehensive Review](https://mlsanity.com/paper/2506.18378/) (2025)
- [Harnessing GPT-4V(ision) for Insurance: A Preliminary Exploration](https://mlsanity.com/paper/2404.09690/) (2024)
- [From large language models to multimodal AI: A scoping review on the potential of generative AI in medicine](https://mlsanity.com/paper/2502.09242/) (2025)
- [Retrieval-augmented in-context learning for multimodal large language models in disease classification](https://mlsanity.com/paper/2505.02087/) (2025)
- [Content Generation Models in Computational Pathology: A Comprehensive Survey on Methods, Applications, and Challenges](https://mlsanity.com/paper/2505.10993/) (2025)
- [Exposing and Mitigating Calibration Biases and Demographic Unfairness in MLLM Few-Shot In-Context Learning for Medical Image Classification](https://mlsanity.com/paper/2506.23298/) (2025)
- [CBVLM: Training-free Explainable Concept-based Large Vision Language Models for Medical Image Classification](https://mlsanity.com/paper/2501.12266/) (2025)
- [Advanced Deep Learning and Large Language Models: Comprehensive Insights for Cancer Detection](https://mlsanity.com/paper/2504.13186/) (2025)
- [Discovering Pathology Rationale and Token Allocation for Efficient Multimodal Pathology Reasoning](https://mlsanity.com/paper/2505.15687/) (2025)
- [The Road to Generalizable Neuro-Symbolic Learning Should be Paved with Foundation Models](https://mlsanity.com/paper/2505.24874/) (2025)
- [Generative Reliability-Based Design Optimization Using In-Context Learning Capabilities of Large Language Models](https://mlsanity.com/paper/2503.22401/) (2025)
- [EgoHandICL: Egocentric 3D Hand Reconstruction with In-Context Learning](https://mlsanity.com/paper/2601.19850/) (2026)
- [BiasICL: In-Context Learning and Demographic Biases of Vision Language Models](https://mlsanity.com/paper/2503.02334/) (2025)
- [Beyond Recognition: Evaluating Visual Perspective Taking in Vision Language Models](https://mlsanity.com/paper/2505.03821/) (2025)
- [MrM: Black-Box Membership Inference Attacks against Multimodal RAG Systems](https://mlsanity.com/paper/2506.07399/) (2025)
- [Smart Eyes for Silent Threats: VLMs and In-Context Learning for THz Imaging](https://mlsanity.com/paper/2507.15576/) (2025)
- [Agentic Design Review System](https://mlsanity.com/paper/2508.10745/) (2025)
- [CG-TTRL: Context-Guided Test-Time Reinforcement Learning for On-Device Large Language Models](https://mlsanity.com/paper/2511.06430/) (2025)
- [Adaptive Diagnostic Reasoning Framework for Pathology with Multimodal Large Language Models](https://mlsanity.com/paper/2511.12008/) (2025)
- [In-Context Learning for Label-Efficient Cancer Image Classification in Oncology](https://mlsanity.com/paper/2505.08798/) (2025)
- [Patient-specific Biomolecular Instruction Tuning](https://mlsanity.com/paper/2509.22853/) (2025)
- [Preliminary Use of Vision Language Model Driven Extraction of Mouse Behavior Towards Understanding Fear Expression](https://mlsanity.com/paper/2510.19160/) (2025)
- [Geometry-Aware Uncertainty Coresets for Robust Visual In-Context Learning in Histopathology](https://mlsanity.com/paper/2605.18419/) (2026)
- [Evaluating Cell Type Inference in Vision Language Models Under Varying Visual Context](https://mlsanity.com/paper/2506.12683/) (2025)
- [PictSure: Pretraining Embeddings Matters for In-Context Learning Image Classifiers](https://mlsanity.com/paper/2506.14842/) (2025)
- [HistoLens: An Interactive XAI Toolkit for Verifying and Mitigating Flaws in Vision-Language Models for Histopathology](https://mlsanity.com/paper/2510.24115/) (2025)
- [FT-ARM: Fine-Tuned Agentic Reflection Multimodal Language Model for Pressure Ulcer Severity Classification with Reasoning](https://mlsanity.com/paper/2510.24980/) (2025)

## Abstract (arXiv)

Medical image classification requires labeled, task-specific datasets which are used to train deep learning networks de novo, or to fine-tune foundation models. However, this process is computationally and technically demanding. In language processing, in-context learning provides an alternative, where models learn from within prompts, bypassing the need for parameter updates. Yet, in-context learning remains underexplored in medical image analysis. Here, we systematically evaluate the model Generative Pretrained Transformer 4 with Vision capabilities (GPT-4V) on cancer image processing with in-context learning on three cancer histopathology tasks of high importance: Classification of tissue subtypes in colorectal cancer, colon polyp subtyping and breast tumor detection in lymph node sections. Our results show that in-context learning is sufficient to match or even outperform specialized neural networks trained for particular tasks, while only requiring a minimal number of samples. In summary, this study demonstrates that large vision language models trained on non-domain specific data can be applied out-of-the box to solve medical image-processing tasks in histopathology. This democratizes access of generalist AI models to medical experts without technical background especially for areas where annotated data is scarce.