mlsanityData to 7 Oct 2026

In-context learning enables multimodal large language models to classify cancer pathology images

arXiv 2403.07407 ↗First posted 12 Mar 2024cs.CVcited by 38

On the map: Multimodal LLMs / vision-language models → Medical vision-language models

Abstract (arXiv)

Medical image classification requires labeled, task-specific datasets which are used to train deep learning networks de novo, or to fine-tune foundation models. However, this process is computationally and technically demanding. In language processing, in-context learning provides an alternative, where models learn from within prompts, bypassing the need for parameter updates. Yet, in-context learning remains underexplored in medical image analysis. Here, we systematically evaluate the model Generative Pretrained Transformer 4 with Vision capabilities (GPT-4V) on cancer image processing with in-context learning on three cancer histopathology tasks of high importance: Classification of tissue subtypes in colorectal cancer, colon polyp subtyping and breast tumor detection in lymph node sections. Our results show that in-context learning is sufficient to match or even outperform specialized neural networks trained for particular tasks, while only requiring a minimal number of samples. In summary, this study demonstrates that large vision language models trained on non-domain specific data can be applied out-of-the box to solve medical image-processing tasks in histopathology. This democratizes access of generalist AI models to medical experts without technical background especially for areas where annotated data is scarce.

Key facts. Cited by 38 papers in the map (data through 7 Oct 2026). Placed in Medical vision-language models, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 20 of its 46 references that are in the map. First posted on arXiv 12 Mar 2024.

Builds on

20 of 46 references.

References · 20

  1. 2020Language Models are Few-Shot Learnerscited by 32,199
  2. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,302
  3. 2023Visual Instruction Tuningcited by 7,792
  4. 2016Grad-CAM: Visual Explanations from Deep Networks via Gradient-based Localizationcited by 6,690
  5. 2022Flamingo: a Visual Language Model for Few-Shot Learningcited by 4,337
  6. 2023Tree of Thoughts: Deliberate Problem Solving with Large Language Modelscited by 3,155
  7. 2022Training Compute-Optimal Large Language Modelscited by 2,927
  8. 2019ZeRO: Memory Optimizations Toward Training Trillion Parameter Modelscited by 1,278
  9. 2023Large Language Models as Optimizerscited by 830
  10. 2020Data Efficient and Weakly Supervised Computational Pathology on Whole Slide Imagescited by 738
  11. 2023The Dawn of LMMs: Preliminary Explorations with GPT-4V(ision)cited by 677
  12. 2023Promptbreeder: Self-Referential Self-Improvement Via Prompt Evolutioncited by 521
8 more
  1. 2020Scaling Laws for Autoregressive Generative Modelingcited by 510
  2. 2023Towards Expert-Level Medical Question Answering with Large Language Modelscited by 382
  3. 2023Can Generalist Foundation Models Outcompete Special-Purpose Tuning? Case Study in Medicinecited by 319
  4. 2022Benchmarking Self-Supervised Learning on Diverse Pathology Datasetscited by 148
  5. 2023Virchow: A Million-Slide Digital Pathology Foundation Modelcited by 86
  6. 2021A Petri Dish for Histopathology Image Analysiscited by 63
  7. 2023A General-Purpose Self-Supervised Model for Computational Pathologycited by 42
  8. 2023From Whole-slide Image to Biomarker Prediction: A Protocol for End-to-End Deep Learning in Computational Pathologycited by 4

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 2; 2025: 27; 2026: 9.

38 papers.

Sort:

By area:

  1. 2025EndoChat: Grounded Multimodal Large Language Model for Endoscopic Surgerycited by 37
  2. 2025True Multimodal In-Context Learning Needs Attention to the Visual Contextcited by 14
  3. 2024A New Era in Computational Pathology: A Survey on Foundation and Vision-Language Modelscited by 9
  4. 2025Taming Vision-Language Models for Medical Image Analysis: A Comprehensive Reviewcited by 9
  5. 2024Harnessing GPT-4V(ision) for Insurance: A Preliminary Explorationcited by 5
  6. 2025From large language models to multimodal AI: A scoping review on the potential of generative AI in medicinecited by 5
  7. 2025Content Generation Models in Computational Pathology: A Comprehensive Survey on Methods, Applications, and Challengescited by 5
  8. 2025Exposing and Mitigating Calibration Biases and Demographic Unfairness in MLLM Few-Shot In-Context Learning for Medical Image Classificationcited by 5
  9. 2025CBVLM: Training-free Explainable Concept-based Large Vision Language Models for Medical Image Classificationcited by 4
  10. 2025Advanced Deep Learning and Large Language Models: Comprehensive Insights for Cancer Detectioncited by 4
  11. 2025Retrieval-augmented in-context learning for multimodal large language models in disease classificationcited by 4
  12. 2025Discovering Pathology Rationale and Token Allocation for Efficient Multimodal Pathology Reasoningcited by 4
26 more
  1. 2025Generative Reliability-Based Design Optimization Using In-Context Learning Capabilities of Large Language Modelscited by 3
  2. 2025The Road to Generalizable Neuro-Symbolic Learning Should be Paved with Foundation Modelscited by 3
  3. 2025BiasICL: In-Context Learning and Demographic Biases of Vision Language Modelscited by 2
  4. 2025Beyond Recognition: Evaluating Visual Perspective Taking in Vision Language Modelscited by 2
  5. 2025MrM: Black-Box Membership Inference Attacks against Multimodal RAG Systemscited by 2
  6. 2025Smart Eyes for Silent Threats: VLMs and In-Context Learning for THz Imagingcited by 2
  7. 2025Agentic Design Review Systemcited by 2
  8. 2025Adaptive Diagnostic Reasoning Framework for Pathology with Multimodal Large Language Modelscited by 2
  9. 2026EgoHandICL: Egocentric 3D Hand Reconstruction with In-Context Learningcited by 2
  10. 2025In-Context Learning for Label-Efficient Cancer Image Classification in Oncologycited by 1
  11. 2025Patient-specific Biomolecular Instruction Tuningcited by 1
  12. 2025Preliminary Use of Vision Language Model Driven Extraction of Mouse Behavior Towards Understanding Fear Expressioncited by 1
  13. 2025CG-TTRL: Context-Guided Test-Time Reinforcement Learning for On-Device Large Language Modelscited by 1
  14. 2025Evaluating Cell Type Inference in Vision Language Models Under Varying Visual Contextcited by 0
  15. 2025PictSure: Pretraining Embeddings Matters for In-Context Learning Image Classifierscited by 0
  16. 2025HistoLens: An Interactive XAI Toolkit for Verifying and Mitigating Flaws in Vision-Language Models for Histopathologycited by 0
  17. 2025FT-ARM: Fine-Tuned Agentic Reflection Multimodal Language Model for Pressure Ulcer Severity Classification with Reasoningcited by 0
  18. 2025In-Context-Learning-Assisted Quality Assessment Vision-Language Models for Metal Additive Manufacturingcited by 0
  19. 2026Transforming Behavioral Neuroscience Discovery with In-Context Learning and AI-Enhanced Tensor Methodscited by 0
  20. 2026Synergizing Discriminative Exemplars and Self-Refined Experience for MLLM-based In-Context Learning in Medical Diagnosiscited by 0
  21. 2026Geometry-Aware Uncertainty Coresets for Robust Visual In-Context Learning in Histopathologycited by 0
  22. 2026Mitigating Batch Effects in Histopathology via Language-Mediated Robust Embedding Generationcited by 0
  23. 2026In-Context Learning for Wound Classification with Small Multimodal Language Modelscited by 0
  24. 2026Performance of large language models in the optical diagnosis of colorectal polypscited by 0
  25. 2026Multimodal LLMs Outperform Pathology Foundation Models in Cross-Domain Histological Similaritycited by 0
  26. 2026How Medical VLMs Underutilize Their Vision Encoders: A Dermatology Perspectivecited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Histopathology Image Report Generation by Vision Language Model with Multimodal In-Context Learning
  2. 2025Investigating Zero-Shot Diagnostic Pathology in Vision-Language Models with Efficient Prompt Design
  3. 2026Representation-guided in-context learning for medical image interpretation with multimodal large language models
  4. 2025Slide-Level Prompt Learning with Vision Language Models for Few-Shot Multiple Instance Learning in Histopathology
  5. 2025Effortless Vision-Language Model Specialization in Histopathology without Annotation
  6. 2025Leveraging Vision-Language Embeddings for Zero-Shot Learning in Histopathology Images
  7. 2024Boosting Vision-Language Models for Histopathology Classification: Predict all at once
  8. 2024Decomposing Disease Descriptions for Enhanced Pathology Detection: A Multi-Aspect Vision-Language Pre-training Framework