mlsanityData to 14 Oct 2026

A Foundational Multimodal Vision Language AI Assistant for Human Pathology

arXiv 2312.07814 ↗First posted 13 Dec 2023cs.CV cs.AIcited by 21

Not on the map. Why?

Abstract (arXiv)

The field of computational pathology has witnessed remarkable progress in the development of both task-specific predictive models and task-agnostic self-supervised vision encoders. However, despite the explosive growth of generative artificial intelligence (AI), there has been limited study on building general purpose, multimodal AI assistants tailored to pathology. Here we present PathChat, a vision-language generalist AI assistant for human pathology using an in-house developed foundational vision encoder pretrained on 100 million histology images from over 100,000 patient cases and 1.18 million pathology image-caption pairs. The vision encoder is then combined with a pretrained large language model and the whole system is finetuned on over 250,000 diverse disease agnostic visual language instructions. We compare PathChat against several multimodal vision language AI assistants as well as GPT4V, which powers the commercially available multimodal general purpose AI assistant ChatGPT-4. When relevant clinical context is provided with the histology image, PathChat achieved a diagnostic accuracy of 87% on multiple-choice questions based on publicly available cases of diverse tissue origins and disease models. Additionally, using open-ended questions and human expert evaluation, we found that overall PathChat produced more accurate and pathologist-preferable responses to diverse queries related to pathology. As an interactive and general vision language AI assistant that can flexibly handle both visual and natural language inputs, PathChat can potentially find impactful applications in pathology education, research, and human-in-the-loop clinical decision making.

Key facts. Cited by 21 papers in the map (data through 14 Oct 2026). Builds on 70 of its 125 references that are in the map. First posted on arXiv 13 Dec 2023.

Builds on

70 of 125 references.

References · 70

  1. 2017Attention Is All You Needcited by 60,361
  2. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  3. 2020Language Models are Few-Shot Learnerscited by 32,419
  4. 2014Microsoft COCO: Common Objects in Contextcited by 17,608
  5. 2022Training language models to follow instructions with human feedbackcited by 15,971
  6. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,599
  7. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 14,132
  8. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 13,028
  9. 2020A Simple Framework for Contrastive Learning of Visual Representationscited by 12,049
  10. 2023DINOv2: Learning Robust Visual Features without Supervisioncited by 7,974
  11. 2023Visual Instruction Tuningcited by 7,869
  12. 2019Momentum Contrast for Unsupervised Visual Representation Learningcited by 7,513
38 more
  1. 2021Masked Autoencoders Are Scalable Vision Learnerscited by 7,403
  2. 2021Emerging Properties in Self-Supervised Vision Transformerscited by 6,968
  3. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,326
  4. 2023Improved Baselines with Visual Instruction Tuningcited by 4,418
  5. 2022Flamingo: a Visual Language Model for Few-Shot Learningcited by 4,386
  6. 2022BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generationcited by 4,322
  7. 2022LAION-5B: An open large-scale dataset for training next generation image-text modelscited by 4,152
  8. 2021Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervisioncited by 3,864
  9. 2023Sparks of Artificial General Intelligence: Early experiments with GPT-4cited by 2,592
  10. 2023A Survey of Large Language Modelscited by 2,576
  11. 2022Emergent Abilities of Large Language Modelscited by 2,512
  12. 2019ViLBERT: Pretraining Task-Agnostic Visiolinguistic Representations for Vision-and-Language Taskscited by 2,222
  13. 2022Large Language Models Encode Clinical Knowledgecited by 1,830
  14. 2021Align before Fuse: Vision and Language Representation Learning with Momentum Distillationcited by 1,619
  15. 2023LLaVA-Med: Training a Large Language-and-Vision Assistant for Biomedicine in One Daycited by 1,305
  16. 2022CoCa: Contrastive Captioners are Image-Text Foundation Modelscited by 1,274
  17. 2020Oscar: Object-Semantics Aligned Pre-training for Vision-Language Taskscited by 1,214
  18. 2023PaLM 2 Technical Reportcited by 1,162
  19. 2021Perceiver: General Perception with Iterative Attentioncited by 1,104
  20. 2019Publicly Available Clinical BERT Embeddingscited by 1,002
  21. 2020Data Efficient and Weakly Supervised Computational Pathology on Whole Slide Imagescited by 745
  22. 2020Contrastive Learning of Medical Visual Representations from Paired Images and Textcited by 641
  23. 2023Capabilities of GPT-4 on Medical Challenge Problemscited by 638
  24. 2022GIT: A Generative Image-to-text Transformer for Vision and Languagecited by 537
  25. 2023BiomedCLIP: a multimodal biomedical foundation model pretrained from fifteen million scientific image-text pairscited by 520
  26. 2022Image as a Foreign Language: BEiT Pretraining for All Vision and Vision-Language Taskscited by 511
  27. 2019Unicoder-VL: A Universal Encoder for Vision and Language by Cross-modal Pre-trainingcited by 451
  28. 2023Med-Flamingo: a Multimodal Medical Few-shot Learnercited by 392
  29. 2018HoVer-Net: Simultaneous Segmentation and Classification of Nuclei in Multi-Tissue Histology Imagescited by 388
  30. 2023Towards Expert-Level Medical Question Answering with Large Language Modelscited by 384
  31. 2022Making the Most of Text Semantics to Improve Biomedical Vision--Language Processingcited by 339
  32. 2023Can Generalist Foundation Models Outcompete Special-Purpose Tuning? Case Study in Medicinecited by 329
  33. 2023Towards Generalist Biomedical AIcited by 289
  34. 2023Multimodal Foundation Models: From Specialists to General-Purpose Assistantscited by 247
  35. 2023PMC-CLIP: Contrastive Language-Image Pre-training using Biomedical Documentscited by 212
  36. 2021Scaling Up Vision-Language Pre-training for Image Captioningcited by 196
  37. 2023Quilt-1M: One Million Image-Text Pairs for Histopathologycited by 194
  38. 2022Benchmarking Self-Supervised Learning on Diverse Pathology Datasetscited by 150

50 most cited of 70 All 70 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 11; 2025: 6; 2026: 4.

21 papers.

Sort:

By area:

  1. 2024Capabilities of Gemini Models in Medicinecited by 237
  2. 2024CARES: A Comprehensive Benchmark of Trustworthiness in Medical Vision Language Modelscited by 64
  3. 2024WSI-LLaVA: A Multimodal Large Language Model for Whole Slide Imagecited by 40
  4. 2025MMedAgent-RL: Optimizing Multi-Agent Collaboration for Multimodal Medical Reasoningcited by 40
  5. 2024Data-Centric Foundation Models in Computational Healthcare: A Surveycited by 27
  6. 2024Interpretable Bilingual Multimodal Large Language Model for Diverse Biomedical Taskscited by 24
  7. 2024Has Multimodal Learning Delivered Universal Intelligence in Healthcare? A Comprehensive Surveycited by 20
  8. 2024GP-VLS: A general-purpose vision language model for surgerycited by 17
  9. 2024Dr-LLaVA: Visual Instruction Tuning with Symbolic Clinical Groundingcited by 10
  10. 2025OctoMed: Data Recipes for State-of-the-Art Multimodal Medical Reasoningcited by 10
  11. 2025LoC-Path: Learning to Compress for Pathology Multimodal Large Language Modelscited by 6
  12. 2026MOOZY: A Patient-First Foundation Model for Computational Pathologycited by 3
9 more
  1. 2024Leveraging Expert Input for Robust and Explainable AI-Assisted Lung Cancer Detection in Chest X-rayscited by 2
  2. 2024From Histopathology Images to Cell Clouds: Learning Slide Representations with Hierarchical Cell Transformercited by 2
  3. 2025Enhancing Multi-task Learning Capability of Medical Generalist Foundation Model via Image-centric Multi-annotation Datacited by 2
  4. 2025The Eye as a Window to Systemic Health: A Survey of Retinal Imaging from Classical Techniques to Oculomicscited by 1
  5. 2026CTIS-QA: Clinical Template-Informed Slide-level Question Answering for Pathologycited by 1
  6. 2024A Survey on Cell Nuclei Instance Segmentation and Classification: Leveraging Context and Attentioncited by 0
  7. 2025The Butterfly Effect in Pathology: Exploring Security in Pathology Foundation Modelscited by 0
  8. 2026Unified Multi-Foundation-Model Slide Representation for Pan-Cancer Recognition and Text-Guided Tumor Localizationcited by 0
  9. 2026CapCLIP: A Vision-Language Representation Alignment Approach for Wireless Capsule Endoscopy Analysiscited by 0