mlsanityData to 7 Oct 2026

MultiMedEval: A Benchmark and a Toolkit for Evaluating Medical Vision-Language Models

arXiv 2402.09262 ↗First posted 14 Feb 2024cs.CVcited by 28

On the map: Multimodal LLMs / vision-language models → Medical vision-language models

Abstract (arXiv)

We introduce MultiMedEval, an open-source toolkit for fair and reproducible evaluation of large, medical vision-language models (VLM). MultiMedEval comprehensively assesses the models' performance on a broad array of six multi-modal tasks, conducted over 23 datasets, and spanning over 11 medical domains. The chosen tasks and performance metrics are based on their widespread adoption in the community and their diversity, ensuring a thorough evaluation of the model's overall generalizability. We open-source a Python toolkit (github.com/corentin-ryr/MultiMedEval) with a simple interface and setup process, enabling the evaluation of any VLM in just a few lines of code. Our goal is to simplify the intricate landscape of VLM evaluation, thus promoting fair and uniform benchmarking of future models.

Key facts. Cited by 28 papers in the map (data through 7 Oct 2026). Placed in Medical vision-language models, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 19 of its 39 references that are in the map. First posted on arXiv 14 Feb 2024.

Builds on

19 of 39 references.

Closest ancestors · 1

Same direction, newest first

  1. 2023BiomedGPT: A Generalist Vision-Language Foundation Model for Diverse Biomedical Tasks26 May

Other references · 18

  1. 2023GPT-4 Technical Reportcited by 18,132
  2. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  3. 2022Flamingo: a Visual Language Model for Few-Shot Learningcited by 4,337
  4. 2019Skin Lesion Analysis Toward Melanoma Detection 2018: A Challenge Hosted by the International Skin Imaging Collaboration (ISIC)cited by 657
  5. 2019The Liver Tumor Segmentation Benchmark (LiTS)cited by 504
  6. 2021SLAKE: A Semantically-Labeled Knowledge-Enhanced Dataset for Medical Visual Question Answeringcited by 444
  7. 2023PMC-VQA: Visual Instruction Tuning for Medical Visual Question Answeringcited by 333
  8. 2020CheXbert: Combining Automatic Labelers and Expert Annotations for Accurate Radiology Report Labeling Using BERTcited by 329
  9. 2021RadGraph: Extracting Clinical Entities and Relations from Radiology Reportscited by 310
  10. 2023Towards Generalist Biomedical AIcited by 286
  11. 2023Towards Generalist Foundation Model for Radiology by Leveraging Web-scale 2D&3D Medical Datacited by 176
  12. 2018Lessons from Natural Language Inference in the Clinical Domaincited by 149
6 more
  1. 2020PAD-UFES-20: a skin lesion dataset composed of patient data and clinical images collected from smartphonescited by 116
  2. 2022VinDr-Mammo: A large-scale benchmark dataset for computer-aided diagnosis in full-field digital mammographycited by 83
  3. 2019Overcoming Data Limitation in Medical Visual Question Answeringcited by 71
  4. 2018The HAM10000 dataset, a large collection of multi-source dermatoscopic images of common pigmented skin lesionscited by 26
  5. 2023RadAdapt: Radiology Report Summarization via Lightweight Domain Adaptation of Large Language Modelscited by 24
  6. 2020Pathological Visual Question Answeringcited by 18

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 7; 2025: 13; 2026: 8.

28 papers.

Sort:

By area:

  1. 2024Advancing Multimodal Medical Capabilities of Geminicited by 88
  2. 2024MMed-RAG: Versatile Multimodal RAG System for Medical Vision Language Modelscited by 84
  3. 2024MME-Survey: A Comprehensive Survey on Evaluation of Multimodal LLMscited by 72
  4. 2024CARES: A Comprehensive Benchmark of Trustworthiness in Medical Vision Language Modelscited by 64
  5. 2024MediConfusion: Can you trust your AI radiologist? Probing the reliability of multimodal medical foundation modelscited by 22
  6. 2024MMedPO: Aligning Medical Vision-Language Models with Clinical-Aware Multimodal Preference Optimizationcited by 21
  7. 2025AgroBench: Vision-Language Model Benchmark in Agriculturecited by 19
  8. 2025DrVD-Bench: Do Vision-Language Models Reason Like Human Doctors in Medical Image Diagnosis?cited by 10
  9. 2025AVA: Attentive VLM Agent for Mastering StarCraft IIcited by 5
  10. 2025SilVar-Med: A Speech-Driven Visual Language Model for Explainable Abnormality Detection in Medical Imagingcited by 5
  11. 20253DReasonKnee: Advancing Grounded Reasoning in Medical Vision Language Modelscited by 5
  12. 2025Uncertainty-Driven Expert Control: Enhancing the Reliability of Medical Vision-Language Modelscited by 4
16 more
  1. 2026MME-CoF-Pro: Evaluating Reasoning Coherence in Video Generative Models with Text and Visual Hintscited by 4
  2. 2025PARC: A Quantitative Framework Uncovering the Symmetries within Vision Language Modelscited by 3
  3. 2025Can Generalist Vision Language Models (VLMs) Rival Specialist Medical VLMs? Benchmarking and Strategic Insightscited by 3
  4. 20256 Fingers, 1 Kidney: Natural Adversarial Medical Images Reveal Critical Weaknesses of Vision-Language Modelscited by 3
  5. 2024Artificial Intelligence for Biomedical Video Generationcited by 2
  6. 2025VSF-Med:A Vulnerability Scoring Framework for Medical Vision-Language Modelscited by 1
  7. 2025MedBLINK: Probing Basic Perception in Multimodal Language Models for Medicinecited by 1
  8. 2025MarineEval: Assessing the Marine Intelligence of Vision-Language Modelscited by 1
  9. 2025Doctor Sun: A Bilingual Multimodal Large Language Model for Biomedical AIcited by 0
  10. 2026Process Supervision for Chain-of-Thought Reasoning via Monte Carlo Net Information Gaincited by 0
  11. 2026When Minor Edits Matter: LLM-Driven Prompt Attack for Medical VLM Robustness in Ultrasoundcited by 0
  12. 2026BUZZY: Contrastive Scoring to Mitigate Text-Induced Bias in Multimodal Multiple-Choice QAcited by 0
  13. 2026CXR-ContraBench: Benchmarking Negated-Option Attraction in Medical VLMscited by 0
  14. 2026Med-R2: An Adversarial Benchmark for Evidence-Grounded Reasoning in Medical VLMscited by 0
  15. 2026EasyLens: A Training-Free Plug-and-Play Subtle-Lesion Representation Amplifier for Medical Vision-Language Modelscited by 0
  16. 2026Authority-Preserving Evaluation of Medical Vision-Language Assistantscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025How Far Have Medical Vision-Language Models Come? A Comprehensive Benchmarking Study
  2. 2026Aloe-Vision: Robust Vision-Language Models for Healthcare
  3. 2025Taming Vision-Language Models for Medical Image Analysis: A Comprehensive Review
  4. 2025EchoBench: Benchmarking Sycophancy in Medical Large Vision-Language Models
  5. 2026How Do Medical MLLMs Fail? A Study on Visual Grounding in Medical Images
  6. 2026Evaluating and Understanding Model Editing for Medical Vision Language Models
  7. 2025LMOD+: A Comprehensive Multimodal Dataset and Benchmark for Developing and Evaluating Multimodal Large Language Models in Ophthalmology
  8. 2026MVC-Bench: Benchmarking Calibration of Medical Vision-Language Models