mlsanityData to 14 Oct 2026

PathAsst: A Generative Foundation AI Assistant Towards Artificial General Intelligence of Pathology

arXiv 2305.15072 ↗First posted 24 May 2023cs.CV cs.MMcited by 57

On the map: Multimodal LLMs / vision-language models → Medical vision-language models

Abstract (arXiv)

As advances in large language models (LLMs) and multimodal techniques continue to mature, the development of general-purpose multimodal large language models (MLLMs) has surged, offering significant applications in interpreting natural images. However, the field of pathology has largely remained untapped, particularly in gathering high-quality data and designing comprehensive model frameworks. To bridge the gap in pathology MLLMs, we present PathAsst, a multimodal generative foundation AI assistant to revolutionize diagnostic and predictive analytics in pathology. The development of PathAsst involves three pivotal steps: data acquisition, CLIP model adaptation, and the training of PathAsst's multimodal generative capabilities. Firstly, we collect over 207K high-quality pathology image-text pairs from authoritative sources. Leveraging the advanced power of ChatGPT, we generate over 180K instruction-following samples. Furthermore, we devise additional instruction-following data specifically tailored for invoking eight pathology-specific sub-models we prepared, allowing the PathAsst to effectively collaborate with these models, enhancing its diagnostic ability. Secondly, by leveraging the collected data, we construct PathCLIP, a pathology-dedicated CLIP, to enhance PathAsst's capabilities in interpreting pathology images. Finally, we integrate PathCLIP with the Vicuna-13b and utilize pathology-specific instruction-tuning data to enhance the multimodal generation capacity of PathAsst and bolster its synergistic interactions with sub-models. The experimental results of PathAsst show the potential of harnessing AI-powered generative foundation model to improve pathology diagnosis and treatment processes.

Key facts. Cited by 57 papers in the map (data through 14 Oct 2026). Placed in Medical vision-language models, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 26 of its 41 references that are in the map. First posted on arXiv 24 May 2023.

Builds on

26 of 41 references.

References · 26

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  2. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  3. 2020Language Models are Few-Shot Learnerscited by 32,419
  4. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,863
  5. 2022Training language models to follow instructions with human feedbackcited by 15,971
  6. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,599
  7. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 14,132
  8. 2023Segment Anythingcited by 8,234
  9. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,326
  10. 2022LAION-5B: An open large-scale dataset for training next generation image-text modelscited by 4,152
  11. 2022OPT: Open Pre-trained Transformer Language Modelscited by 3,512
  12. 2017Billion-scale similarity search with GPUscited by 2,497
14 more
  1. 2023PaLM-E: An Embodied Multimodal Language Modelcited by 2,071
  2. 2021Conceptual 12M: Pushing Web-Scale Image-Text Pre-Training To Recognize Long-Tail Visual Conceptscited by 1,216
  3. 2020Domain-Specific Language Model Pretraining for Biomedical Natural Language Processingcited by 1,009
  4. 2022YOLOv7: Trainable bag-of-freebies sets new state-of-the-art for real-time object detectorscited by 982
  5. 2023LLaMA-Adapter V2: Parameter-Efficient Visual Instruction Modelcited by 602
  6. 2023MM-REACT: Prompting ChatGPT for Multimodal Reasoning and Actioncited by 517
  7. 2020PathVQA: 30000+ Questions for Medical Visual Question Answeringcited by 475
  8. 2023PMC-CLIP: Contrastive Language-Image Pre-training using Biomedical Documentscited by 212
  9. 2019Lung and Colon Cancer Histopathological Image Dataset (LC25000)cited by 89
  10. 2020MedICaT: A Dataset of Medical Images, Captions, and Textual Referencescited by 71
  11. 2023Open-Ended Medical Visual Question Answering Through Prefix Tuning of Language Modelscited by 58
  12. 2021Multiple Meta-model Quantifying for Medical Visual Question Answeringcited by 54
  13. 2022WSSS4LUAD: Grand Challenge on Weakly-supervised Tissue Semantic Segmentation for Lung Adenocarcinomacited by 36
  14. 2021Investigating Pose Representations and Motion Contexts Modeling for 3D Motion Predictioncited by 31

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 8; 2024: 13; 2025: 18; 2026: 18.

57 papers.

Sort:

By area:

  1. 2023Multimodal Foundation Models: From Specialists to General-Purpose Assistantscited by 247
  2. 2024MMedAgent: Learning to Use Medical Tools with Multi-modal Agentcited by 109
  3. 2023Embodied Task Planning with Large Language Modelscited by 84
  4. 2024Foundation Model for Advancing Healthcare: Challenges, Opportunities, and Future Directionscited by 77
  5. 2024SlideChat: A Large Vision-Language Assistant for Whole-Slide Pathology Image Understandingcited by 57
  6. 2024A Comprehensive Survey of Large Language Models and Multimodal Large Language Models in Medicinecited by 45
  7. 2024CPath-Omni: A Unified Multimodal Foundation Model for Patch and Whole Slide Image Analysis in Computational Pathologycited by 34
  8. 2024PathAlign: A vision-language model for whole slide images in histopathologycited by 30
  9. 2025CPathAgent: An Agent-based Foundation Model for Interpretable High-Resolution Pathology Image Analysis Mimicking Pathologists' Diagnostic Logiccited by 29
  10. 2024Foundation Models for Biomedical Image Segmentation: A Surveycited by 28
  11. 2024PathGen-1.6M: 1.6 Million Pathology Image-text Pairs Generation through Multi-agent Collaborationcited by 28
  12. 2025Evidence-based diagnostic reasoning with multi-agent copilot for human pathologycited by 27
38 more
  1. 2023MarineGPT: Unlocking Secrets of Ocean to the Publiccited by 23
  2. 2023A Foundational Multimodal Vision Language AI Assistant for Human Pathologycited by 21
  3. 2024Has Multimodal Learning Delivered Universal Intelligence in Healthcare? A Comprehensive Surveycited by 20
  4. 2023Large Multimodal Models: Notes on CVPR 2023 Tutorialcited by 18
  5. 2025A Survey on Computational Pathology Foundation Models: Datasets, Adaptation Strategies, and Evaluation Taskscited by 16
  6. 2025Multimodal Large Language Models for Medicine: A Comprehensive Surveycited by 16
  7. 2023Multimodal Foundation Models for Zero-shot Animal Species Recognition in Camera Trap Imagescited by 13
  8. 2023Path to Medical AGI: Unify Domain-specific Medical LLMs with the Lowest Costcited by 12
  9. 2025A Versatile Pathology Co-pilot via Reasoning Enhanced Multimodal Large Language Modelcited by 12
  10. 2024A New Era in Computational Pathology: A Survey on Foundation and Vision-Language Modelscited by 9
  11. 2025Multi-Modal Foundation Models for Computational Pathology: A Surveycited by 9
  12. 2025PathDiff: Histopathology Image Synthesis with Unpaired Text and Mask Conditionscited by 9
  13. 2025Foundation Models in Medical Imaging: A Review and Outlookcited by 8
  14. 2025Content Generation Models in Computational Pathology: A Comprehensive Survey on Methods, Applications, and Challengescited by 5
  15. 2025Discovering Pathology Rationale and Token Allocation for Efficient Multimodal Pathology Reasoningcited by 4
  16. 2026LAMMI-Pathology: A Tool-Centric Bottom-Up LVLM-Agent Framework for Molecularly Informed Medical Intelligence in Pathologycited by 3
  17. 2026MLLM-HWSI: A Multimodal Large Language Model for Hierarchical Whole Slide Image Understandingcited by 3
  18. 2024Rapid Review of Generative AI in Smart Medical Applicationscited by 2
  19. 2025Histopathology Multi-modal Embedding for Pathology Composed Retrievalcited by 2
  20. 2025Enhancing Multi-task Learning Capability of Medical Generalist Foundation Model via Image-centric Multi-annotation Datacited by 2
  21. 2026Do Pathology Vision-Language Models Truly See Pathology?cited by 2
  22. 2023Evaluating the Robustness to Instructions of Large Language Modelscited by 1
  23. 2025PhenoBench: A Comprehensive Benchmark for Cell Phenotypingcited by 1
  24. 2025HeteroRAG: A Heterogeneous Retrieval-Augmented Generation Framework for Medical Vision Language Taskscited by 1
  25. 2025Patient-specific Biomolecular Instruction Tuningcited by 1
  26. 2026ReinPath: A Multimodal Reinforcement Learning Approach for Pathologycited by 1
  27. 2026PathMem: Toward Cognition-Aligned Memory Transformation for Pathology MLLMscited by 1
  28. 2026HiPath: Hierarchical Vision-Language Alignment for Structured Pathology Report Predictioncited by 1
  29. 2026LLaVA-LE: Large Language-and-Vision Assistant for Lunar Explorationcited by 1
  30. 2026Agentic Visual Reasoning in Whole-Slide Pathology Images via Active Perceptioncited by 1
  31. 2024Efficient and Comprehensive Feature Extraction in Large Vision-Language Model for Pathology Analysiscited by 0
  32. 2024Implementing Trust in Non-Small Cell Lung Cancer Diagnosis with a Conformalized Uncertainty-Aware AI Framework in Whole-Slide Imagescited by 0
  33. 2025VLM-based Prompts as the Optimal Assistant for Unpaired Histopathology Virtual Stainingcited by 0
  34. 2025ViDRiP-LLaVA: A Dataset and Benchmark for Diagnostic Reasoning from Pathology Videoscited by 0
  35. 2025DiagR1: A Vision-Language Model Trained via Reinforcement Learning for Digestive Pathology Diagnosiscited by 0
  36. 2026Layout-Guided Controllable Pathology Image Generation with In-Context Diffusion Transformerscited by 0
  37. 2026SSMamba: A Self-Supervised Hybrid State Space Model for Pathological Image Classificationcited by 0
  38. 2026PBSBench: A Multi-Level Vision-Language Framework and Benchmark for Hematopathology Whole Slide Image Interpretationcited by 0

50 most cited of 57 All 57 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2024PathMMU: A Massive Multimodal Expert-Level Benchmark for Understanding and Reasoning in Pathology
  2. 2025PathMR: Multimodal Visual Reasoning for Interpretable Pathology Diagnosis
  3. 2025Patho-R1: A Multimodal Reinforcement Learning-Based Pathology Expert Reasoner
  4. 2025PathoHR: Hierarchical Reasoning for Vision-Language Models in Pathology
  5. 2025TeamPath: Building MultiModal Pathology Experts with Reasoning AI Copilots
  6. 2025MPath: Multimodal Pathology Report Generation from Whole Slide Images
  7. 2026PathView-Bench: Can Multimodal Large Language Models Achieve Fine-grained Multiscale Understanding of Pathology Images?
  8. 2025Adaptive Diagnostic Reasoning Framework for Pathology with Multimodal Large Language Models