# Gemini Goes to Med School: Exploring the Capabilities of Multimodal Large Language Models on Medical Challenge Problems & Hallucinations

arXiv: https://arxiv.org/abs/2402.07023 · HTML: https://mlsanity.com/paper/2402.07023/

## Key facts

- Cited by 27 papers in the map (data through 14 Oct 2026).
- Placed in Hallucination mitigation in LVLMs, one of 16 directions in Multimodal LLMs / vision-language models.
- Builds on 14 of its 27 references that are in the map.
- First posted on arXiv 10 Feb 2024.

## Placed in

- Multimodal LLMs / vision-language models / [Hallucination mitigation in LVLMs](https://mlsanity.com/mllm/hallucination-mitigation-in-lvlms/)

## Builds on (most cited first, 14 of 14)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [Measuring Massive Multitask Language Understanding](https://mlsanity.com/paper/2009.03300/) (2020)
- [PaLM: Scaling Language Modeling with Pathways](https://mlsanity.com/paper/2204.02311/) (2022)
- [The Curious Case of Neural Text Degeneration](https://mlsanity.com/paper/1904.09751/) (2019)
- [Code Llama: Open Foundation Models for Code](https://mlsanity.com/paper/2308.12950/) (2023)
- [Large Language Models Encode Clinical Knowledge](https://mlsanity.com/paper/2212.13138/) (2022)
- [What Disease does this Patient Have? A Large-scale Open Domain Question Answering Dataset from Medical Exams](https://mlsanity.com/paper/2009.13081/) (2020)
- [PubMedQA: A Dataset for Biomedical Research Question Answering](https://mlsanity.com/paper/1909.06146/) (2019)
- [Capabilities of GPT-4 on Medical Challenge Problems](https://mlsanity.com/paper/2303.13375/) (2023)
- [Zephyr: Direct Distillation of LM Alignment](https://mlsanity.com/paper/2310.16944/) (2023)
- [Towards Understanding Chain-of-Thought Prompting: An Empirical Study of What Matters](https://mlsanity.com/paper/2212.10001/) (2022)
- [Can large language models reason about medical questions?](https://mlsanity.com/paper/2207.08143/) (2022)
- [Med-HALT: Medical Domain Hallucination Test for Large Language Models](https://mlsanity.com/paper/2307.15343/) (2023)

## Built on by (most cited first, 27 of 27)

- [Advancing Multimodal Medical Capabilities of Gemini](https://mlsanity.com/paper/2405.03162/) (2024)
- [The State of Multilingual LLM Safety Research: From Measuring the Language Gap to Mitigating It](https://mlsanity.com/paper/2505.24119/) (2025)
- [ErrorRadar: Benchmarking Complex Mathematical Reasoning of Multimodal Large Language Models Via Error Detection](https://mlsanity.com/paper/2410.04509/) (2024)
- [Securing Large Language Models: Addressing Bias, Misinformation, and Prompt Attacks](https://mlsanity.com/paper/2409.08087/) (2024)
- [Fact-Aware Multimodal Retrieval Augmentation for Accurate Medical Radiology Report Generation](https://mlsanity.com/paper/2407.15268/) (2024)
- [Has Multimodal Learning Delivered Universal Intelligence in Healthcare? A Comprehensive Survey](https://mlsanity.com/paper/2408.12880/) (2024)
- [ReXVQA: A Large-scale Visual Question Answering Benchmark for Generalist Chest X-ray Understanding](https://mlsanity.com/paper/2506.04353/) (2025)
- [Virtual Agents for Alcohol Use Counseling: Exploring LLM-Powered Motivational Interviewing](https://mlsanity.com/paper/2407.08095/) (2024)
- [Hippocrates: An Open-Source Framework for Advancing Large Language Models in Healthcare](https://mlsanity.com/paper/2404.16621/) (2024)
- [From Text to Multimodality: Exploring the Evolution and Impact of Large Language Models in Medical Practice](https://mlsanity.com/paper/2410.01812/) (2024)
- [Reliable and diverse evaluation of LLM medical knowledge mastery](https://mlsanity.com/paper/2409.14302/) (2024)
- [An evaluation of DeepSeek Models in Biomedical Natural Language Processing](https://mlsanity.com/paper/2503.00624/) (2025)
- [NPHardEval4V: Dynamic Evaluation of Large Vision-Language Models with Effects of Vision](https://mlsanity.com/paper/2403.01777/) (2024)
- [Faculty Perspectives on the Potential of RAG in Computer Science Higher Education](https://mlsanity.com/paper/2408.01462/) (2024)
- [Image First or Text First? Optimising the Sequencing of Modalities in Large Language Model Prompting and Reasoning Tasks](https://mlsanity.com/paper/2410.03062/) (2024)
- [MARK: Memory Augmented Refinement of Knowledge](https://mlsanity.com/paper/2505.05177/) (2025)
- [Establishing a Scale for Kullback-Leibler Divergence in Language Models Across Various Settings](https://mlsanity.com/paper/2505.15353/) (2025)
- [Pareto-Optimized Open-Source LLMs for Healthcare via Context Retrieval](https://mlsanity.com/paper/2409.15127/) (2024)
- [A Scoping Review of Natural Language Processing in Addressing Medically Inaccurate Information: Errors, Misinformation, and Hallucination](https://mlsanity.com/paper/2505.00008/) (2025)
- [Tonguescape: Exploring Language Models Understanding of Vowel Articulation](https://mlsanity.com/paper/2501.17643/) (2025)
- [Generative Artificial Intelligence-Guided User Studies: An Application for Air Taxi Services](https://mlsanity.com/paper/2406.12296/) (2024)
- [Generative Language Models Potential for Requirement Engineering Applications: Insights into Current Strengths and Limitations](https://mlsanity.com/paper/2412.00959/) (2024)
- [Medalyze: Lightweight Medical Report Summarization Application Using FLAN-T5-Large](https://mlsanity.com/paper/2505.17059/) (2025)
- [OR-VSKC: Resolving Visual-Semantic Knowledge Conflicts in Operating Rooms with Synthetic Data-Guided Alignment](https://mlsanity.com/paper/2506.22500/) (2025)
- [Investigating LLM Capabilities on Long Context Comprehension for Medical Question Answering](https://mlsanity.com/paper/2510.18691/) (2025)
- [BUSTR: Descriptor-Aware Vision-Language Learning for Breast Ultrasound Report Generation](https://mlsanity.com/paper/2511.20956/) (2025)
- [Toward Automatic Safe Driving Instruction: A Large-Scale Vision Language Model Approach](https://mlsanity.com/paper/2511.23311/) (2025)

## Abstract (arXiv)

Large language models have the potential to be valuable in the healthcare industry, but it's crucial to verify their safety and effectiveness through rigorous evaluation. For this purpose, we comprehensively evaluated both open-source LLMs and Google's new multimodal LLM called Gemini across Medical reasoning, hallucination detection, and Medical Visual Question Answering tasks. While Gemini showed competence, it lagged behind state-of-the-art models like MedPaLM 2 and GPT-4 in diagnostic accuracy. Additionally, Gemini achieved an accuracy of 61.45% on the medical VQA dataset, significantly lower than GPT-4V's score of 88%. Our analysis revealed that Gemini is highly susceptible to hallucinations, overconfidence, and knowledge gaps, which indicate risks if deployed uncritically. We also performed a detailed analysis by medical subject and test type, providing actionable feedback for developers and clinicians. To mitigate risks, we applied prompting strategies that improved performance. Additionally, we facilitated future research and development by releasing a Python module for medical LLM evaluation and establishing a dedicated leaderboard on Hugging Face for medical domain LLMs. Python module can be found at https://github.com/promptslab/RosettaEval