mlsanityData to 7 Oct 2026

JMed48k: A Multi-Profession Japanese Medical Licensing Benchmark for Vision-Language Model Evaluation

arXiv 2605.22080 ↗First posted 21 May 2026cs.CV cs.AIcited by 0

On the map: Multimodal LLMs / vision-language models → Medical vision-language models

Abstract (arXiv)

We introduce JMed48k, a multi-profession Japanese healthcare licensing benchmark for evaluating vision-language models. Built from official PDF materials released by the Japanese Ministry of Health, Labour and Welfare, JMed48k contains 48,862 exam questions and 20,142 images from 11 national licensing examinations between 2005 and 2025, with visual content annotated under an 8-type taxonomy. From this corpus, we derive JMed48k-Eval, a recent five-year evaluation subset with 12,484 scored questions, including 9,905 text-only questions and 2,579 questions with images. We evaluate 21 proprietary, open-source, and medical-specific models, reporting text-only and with-image performance separately. Because these subsets contain different questions, we further introduce a paired image-removal audit that evaluates questions with images before and after removing visual content to explore four answer-transition states. The audit shows that proprietary and open source models gain substantially from images, whereas medical-specific systems show limited observable use of visual evidence, with many correct answers persisting after image removal. Even among proprietary models, the net image-removal effect varies sevenfold across professions, from +5.7 points on Physician questions to +39.8 points on Public Health Nurse questions. We release JMed48k to support reproducible, profession-stratified evaluation of vision-language models in medical licensing settings.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Placed in Medical vision-language models, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 17 of its 56 references that are in the map. First posted on arXiv 21 May 2026.

Builds on

17 of 56 references.

Closest ancestors · 2

Same direction, newest first

  1. 2026KorMedMCQA-V: A Multimodal Benchmark for Evaluating Vision-Language Models on the Korean Medical Licensing Examination14 Feb
  2. 2025Lingshu: A Generalist Foundation Model for Unified Multimodal Medical Understanding and Reasoning8 Jun

Other references · 15

  1. 2025DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learningcited by 6,146
  2. 2025Gemini 2.5: Pushing the Frontier with Advanced Reasoning, Multimodality, Long Context, and Next Generation Agentic Capabilitiescited by 3,655
  3. 2025Qwen3-VL Technical Reportcited by 2,534
  4. 2023MME: A Comprehensive Evaluation Benchmark for Multimodal Large Language Modelscited by 1,624
  5. 2025OpenAI GPT-5 System Cardcited by 910
  6. 2023Capabilities of GPT-4 on Medical Challenge Problemscited by 630
  7. 2020PathVQA: 30000+ Questions for Medical Visual Question Answeringcited by 467
  8. 2025GLM-4.5V and GLM-4.1V-Thinking: Towards Versatile Multimodal Reasoning with Scalable Reinforcement Learningcited by 439
  9. 2025MedGemma Technical Reportcited by 398
  10. 2023PMC-VQA: Visual Instruction Tuning for Medical Visual Question Answeringcited by 333
  11. 2024MinerU: An Open-Source Solution for Precise Document Content Extractioncited by 255
  12. 2023Towards Generalist Foundation Model for Radiology by Leveraging Web-scale 2D&3D Medical Datacited by 176
3 more
  1. 2025Hulu-Med: A Transparent Generalist Model towards Holistic Medical Vision-Language Understandingcited by 96
  2. 2023Evaluating GPT-4 and ChatGPT on Japanese Medical Licensing Examinationscited by 63
  3. 2024KorMedMCQA: Multi-Choice Question Answering Benchmark for Korean Healthcare Professional Licensing Examinationscited by 22

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2025KokushiMD-10: Benchmark for Evaluating Large Language Models on Ten Japanese National Healthcare Licensing Examinations
  2. 2026Polish Medical Visual Question Answering: Vision-Language Models Underutilize Visual Evidence
  3. 2024MultiMedEval: A Benchmark and a Toolkit for Evaluating Medical Vision-Language Models
  4. 2025How Far Have Medical Vision-Language Models Come? A Comprehensive Benchmarking Study
  5. 2024WorldMedQA-V: a multilingual, multimodal medical examination dataset for multimodal language models evaluation
  6. 2024EXAMS-V: A Multi-Discipline Multilingual Multimodal Exam Benchmark for Evaluating Vision Language Models
  7. 2024Medical Adaptation of Large Language and Vision-Language Models: Are We Making Progress?
  8. 2026A Controlled Audit of Pretraining Contamination in Public Medical Vision-Language Benchmarks