# PathMMU: A Massive Multimodal Expert-Level Benchmark for Understanding and Reasoning in Pathology

arXiv: https://arxiv.org/abs/2401.16355 · HTML: https://mlsanity.com/paper/2401.16355/

## Key facts

- Cited by 43 papers in the map (data through 14 Oct 2026).
- Placed in Medical vision-language models, one of 16 directions in Multimodal LLMs / vision-language models.
- Builds on 41 of its 58 references that are in the map.
- First posted on arXiv 29 Jan 2024.

## Placed in

- Multimodal LLMs / vision-language models / [Medical vision-language models](https://mlsanity.com/mllm/medical-vision-language-models/)

## Builds on (most cited first, 30 of 41)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer](https://mlsanity.com/paper/1910.10683/) (2019)
- [LLaMA: Open and Efficient Foundation Language Models](https://mlsanity.com/paper/2302.13971/) (2023)
- [Visual Instruction Tuning](https://mlsanity.com/paper/2304.08485/) (2023)
- [BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Models](https://mlsanity.com/paper/2301.12597/) (2023)
- [Flamingo: a Visual Language Model for Few-Shot Learning](https://mlsanity.com/paper/2204.14198/) (2022)
- [A ConvNet for the 2020s](https://mlsanity.com/paper/2201.03545/) (2022)
- [InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuning](https://mlsanity.com/paper/2305.06500/) (2023)
- [MiniGPT-4: Enhancing Vision-Language Understanding with Advanced Large Language Models](https://mlsanity.com/paper/2304.10592/) (2023)
- [MMBench: Is Your Multi-modal Model an All-around Player?](https://mlsanity.com/paper/2307.06281/) (2023)
- [MMMU: A Massive Multi-discipline Multimodal Understanding and Reasoning Benchmark for Expert AGI](https://mlsanity.com/paper/2311.16502/) (2023)
- [PaLM-E: An Embodied Multimodal Language Model](https://mlsanity.com/paper/2303.03378/) (2023)
- [LLaVA-Med: Training a Large Language-and-Vision Assistant for Biomedicine in One Day](https://mlsanity.com/paper/2306.00890/) (2023)
- [MM-Vet: Evaluating Large Multimodal Models for Integrated Capabilities](https://mlsanity.com/paper/2308.02490/) (2023)
- [Kosmos-2: Grounding Multimodal Large Language Models to the World](https://mlsanity.com/paper/2306.14824/) (2023)
- [Domain-Specific Language Model Pretraining for Biomedical Natural Language Processing](https://mlsanity.com/paper/2007.15779/) (2020)
- [SEED-Bench: Benchmarking Multimodal LLMs with Generative Comprehension](https://mlsanity.com/paper/2307.16125/) (2023)
- [CogVLM: Visual Expert for Pretrained Language Models](https://mlsanity.com/paper/2311.03079/) (2023)
- [LLaMA-Adapter V2: Parameter-Efficient Visual Instruction Model](https://mlsanity.com/paper/2304.15010/) (2023)
- [HallusionBench: An Advanced Diagnostic Suite for Entangled Language Hallucination and Visual Illusion in Large Vision-Language Models](https://mlsanity.com/paper/2310.14566/) (2023)
- [Otter: A Multi-Modal Model with In-Context Instruction Tuning](https://mlsanity.com/paper/2305.03726/) (2023)
- [PathVQA: 30000+ Questions for Medical Visual Question Answering](https://mlsanity.com/paper/2003.10286/) (2020)
- [Rotation Equivariant CNNs for Digital Pathology](https://mlsanity.com/paper/1806.03962/) (2018)
- [PMC-VQA: Visual Instruction Tuning for Medical Visual Question Answering](https://mlsanity.com/paper/2305.10415/) (2023)
- [YOLOv6: A Single-Stage Object Detection Framework for Industrial Applications](https://mlsanity.com/paper/2209.02976/) (2022)
- [ERNIE 3.0: Large-scale Knowledge Enhanced Pre-training for Language Understanding and Generation](https://mlsanity.com/paper/2107.02137/) (2021)
- [LVLM-eHub: A Comprehensive Evaluation Benchmark for Large Vision-Language Models](https://mlsanity.com/paper/2306.09265/) (2023)
- [Quilt-1M: One Million Image-Text Pairs for Histopathology](https://mlsanity.com/paper/2306.11207/) (2023)

## Built on by (most cited first, 30 of 43)

- [GMAI-MMBench: A Comprehensive Multimodal Evaluation Benchmark Towards General Medical AI](https://mlsanity.com/paper/2408.03361/) (2024)
- [PathAlign: A vision-language model for whole slide images in histopathology](https://mlsanity.com/paper/2406.19578/) (2024)
- [CPathAgent: An Agent-based Foundation Model for Interpretable High-Resolution Pathology Image Analysis Mimicking Pathologists' Diagnostic Logic](https://mlsanity.com/paper/2505.20510/) (2025)
- [Holistic Evaluation of Multimodal LLMs on Spatial Intelligence](https://mlsanity.com/paper/2508.13142/) (2025)
- [PathGen-1.6M: 1.6 Million Pathology Image-text Pairs Generation through Multi-agent Collaboration](https://mlsanity.com/paper/2407.00203/) (2024)
- [Evidence-based diagnostic reasoning with multi-agent copilot for human pathology](https://mlsanity.com/paper/2506.20964/) (2025)
- [Towards Better Dental AI: A Multimodal Benchmark and Instruction Dataset for Panoramic X-ray Analysis](https://mlsanity.com/paper/2509.09254/) (2025)
- [Patho-AgenticRAG: Towards Multimodal Agentic Retrieval-Augmented Generation for Pathology VLMs via Reinforcement Learning](https://mlsanity.com/paper/2508.02258/) (2025)
- [A Versatile Pathology Co-pilot via Reasoning Enhanced Multimodal Large Language Model](https://mlsanity.com/paper/2507.17303/) (2025)
- [DrVD-Bench: Do Vision-Language Models Reason Like Human Doctors in Medical Image Diagnosis?](https://mlsanity.com/paper/2505.24173/) (2025)
- [A New Era in Computational Pathology: A Survey on Foundation and Vision-Language Models](https://mlsanity.com/paper/2408.14496/) (2024)
- [Multi-Modal Foundation Models for Computational Pathology: A Survey](https://mlsanity.com/paper/2503.09091/) (2025)
- [Navigating Gigapixel Pathology Images with Large Multimodal Models](https://mlsanity.com/paper/2511.19652/) (2025)
- [PathNavigate: A Training-Free Pathology Agent with Surprise-Guided Scan and Shared Slide Memory for Whole-Slide Image VQA](https://mlsanity.com/paper/2605.23559/) (2026)
- [MedMax: Mixed-Modal Instruction Tuning for Training Biomedical Assistants](https://mlsanity.com/paper/2412.12661/) (2024)
- [Content Generation Models in Computational Pathology: A Comprehensive Survey on Methods, Applications, and Challenges](https://mlsanity.com/paper/2505.10993/) (2025)
- [TeamPath: Building MultiModal Pathology Experts with Reasoning AI Copilots](https://mlsanity.com/paper/2511.17652/) (2025)
- [Discovering Pathology Rationale and Token Allocation for Efficient Multimodal Pathology Reasoning](https://mlsanity.com/paper/2505.15687/) (2025)
- [MDK12-Bench: A Comprehensive Evaluation of Multimodal Large Language Models on Multidisciplinary Exams](https://mlsanity.com/paper/2508.06851/) (2025)
- [Beyond Pixel Simulation: Pathology Image Generation via Diagnostic Semantic Tokens and Prototype Control](https://mlsanity.com/paper/2512.21058/) (2025)
- [Instance-Aligned Captions for Explainable Video Anomaly Detection](https://mlsanity.com/paper/2601.08155/) (2026)
- [Domain Specific Benchmarks for Evaluating Multimodal Large Language Models](https://mlsanity.com/paper/2506.12958/) (2025)
- [LAMMI-Pathology: A Tool-Centric Bottom-Up LVLM-Agent Framework for Molecularly Informed Medical Intelligence in Pathology](https://mlsanity.com/paper/2602.18773/) (2026)
- [Enhancing Pathological VLMs with Cross-scale Reasoning](https://mlsanity.com/paper/2606.17412/) (2026)
- [Histopathology Multi-modal Embedding for Pathology Composed Retrieval](https://mlsanity.com/paper/2502.07221/) (2025)
- [PathVG: A New Benchmark and Dataset for Pathology Visual Grounding](https://mlsanity.com/paper/2502.20869/) (2025)
- [Benchmarking PhD-Level Coding in 3D Geometric Computer Vision](https://mlsanity.com/paper/2603.30038/) (2026)
- [Beyond a Single Frame: Multi-Frame Spatially Grounded Reasoning Across Volumetric MRI](https://mlsanity.com/paper/2604.15808/) (2026)
- [Do Pathology Vision-Language Models Truly See Pathology?](https://mlsanity.com/paper/2607.21065/) (2026)
- [HeteroRAG: A Heterogeneous Retrieval-Augmented Generation Framework for Medical Vision Language Tasks](https://mlsanity.com/paper/2508.12778/) (2025)

## Abstract (arXiv)

The emergence of large multimodal models has unlocked remarkable potential in AI, particularly in pathology. However, the lack of specialized, high-quality benchmark impeded their development and precise evaluation. To address this, we introduce PathMMU, the largest and highest-quality expert-validated pathology benchmark for Large Multimodal Models (LMMs). It comprises 33,428 multimodal multi-choice questions and 24,067 images from various sources, each accompanied by an explanation for the correct answer. The construction of PathMMU harnesses GPT-4V's advanced capabilities, utilizing over 30,000 image-caption pairs to enrich captions and generate corresponding Q&As in a cascading process. Significantly, to maximize PathMMU's authority, we invite seven pathologists to scrutinize each question under strict standards in PathMMU's validation and test sets, while simultaneously setting an expert-level performance benchmark for PathMMU. We conduct extensive evaluations, including zero-shot assessments of 14 open-sourced and 4 closed-sourced LMMs and their robustness to image corruption. We also fine-tune representative LMMs to assess their adaptability to PathMMU. The empirical findings indicate that advanced LMMs struggle with the challenging PathMMU benchmark, with the top-performing LMM, GPT-4V, achieving only a 49.8% zero-shot performance, significantly lower than the 71.8% demonstrated by human pathologists. After fine-tuning, significantly smaller open-sourced LMMs can outperform GPT-4V but still fall short of the expertise shown by pathologists. We hope that the PathMMU will offer valuable insights and foster the development of more specialized, next-generation LMMs for pathology.