mlsanityData to 14 Oct 2026

Capabilities of Gemini Models in Medicine

arXiv 2404.18416 ↗First posted 29 Apr 2024cs.AI cs.CL cs.CVcited by 237

Not on the map. Why?

Cited by 38 in LLM reasoning, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

Excellence in a wide variety of medical applications poses considerable challenges for AI, requiring advanced reasoning, access to up-to-date medical knowledge and understanding of complex multimodal data. Gemini models, with strong general capabilities in multimodal and long-context reasoning, offer exciting possibilities in medicine. Building on these core strengths of Gemini, we introduce Med-Gemini, a family of highly capable multimodal models that are specialized in medicine with the ability to seamlessly use web search, and that can be efficiently tailored to novel modalities using custom encoders. We evaluate Med-Gemini on 14 medical benchmarks, establishing new state-of-the-art (SoTA) performance on 10 of them, and surpass the GPT-4 model family on every benchmark where a direct comparison is viable, often by a wide margin. On the popular MedQA (USMLE) benchmark, our best-performing Med-Gemini model achieves SoTA performance of 91.1% accuracy, using a novel uncertainty-guided search strategy. On 7 multimodal benchmarks including NEJM Image Challenges and MMMU (health & medicine), Med-Gemini improves over GPT-4V by an average relative margin of 44.5%. We demonstrate the effectiveness of Med-Gemini's long-context capabilities through SoTA performance on a needle-in-a-haystack retrieval task from long de-identified health records and medical video question answering, surpassing prior bespoke methods using only in-context learning. Finally, Med-Gemini's performance suggests real-world utility by surpassing human experts on tasks such as medical text summarization, alongside demonstrations of promising potential for multimodal medical dialogue, medical research and education. Taken together, our results offer compelling evidence for Med-Gemini's potential, although further rigorous evaluation will be crucial before real-world deployment in this safety-critical domain.

Key facts. Cited by 237 papers in the map (data through 14 Oct 2026). Builds on 42 of its 148 references that are in the map. First posted on arXiv 29 Apr 2024.

Builds on

42 of 148 references.

References · 42

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  2. 2023GPT-4 Technical Reportcited by 18,396
  3. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 14,132
  4. 2022Self-Consistency Improves Chain of Thought Reasoning in Language Modelscited by 5,708
  5. 2021Finetuned Language Models Are Zero-Shot Learnerscited by 3,780
  6. 2017Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layercited by 3,509
  7. 2023MMMU: A Massive Multi-discipline Multimodal Understanding and Reasoning Benchmark for Expert AGIcited by 2,080
  8. 2023PaLM-E: An Embodied Multimodal Language Modelcited by 2,071
  9. 2023ToolLLM: Facilitating Large Language Models to Master 16000+ Real-world APIscited by 1,947
  10. 2019Transformer-XL: Attentive Language Models Beyond a Fixed-Length Contextcited by 1,850
  11. 2021WebGPT: Browser-assisted question-answering with human feedbackcited by 1,745
  12. 2022CoCa: Contrastive Captioners are Image-Text Foundation Modelscited by 1,274
30 more
  1. 2023PaLM 2 Technical Reportcited by 1,162
  2. 2023Gemini: A Family of Highly Capable Multimodal Modelscited by 872
  3. 2022PaLI: A Jointly-Scaled Multilingual Language-Image Modelcited by 766
  4. 2019MIMIC-CXR-JPG, a large publicly available database of labeled chest radiographscited by 597
  5. 2023Towards Expert-Level Medical Question Answering with Large Language Modelscited by 384
  6. 2023Can Generalist Foundation Models Outcompete Special-Purpose Tuning? Case Study in Medicinecited by 329
  7. 2023Large Language Models as Tool Makerscited by 303
  8. 2024RAFT: Adapting Language Model to Domain Specific RAGcited by 262
  9. 2023A Stitch in Time Saves Nine: Detecting and Mitigating Hallucinations of LLMs by Validating Low-Confidence Generationcited by 206
  10. 2023Med-HALT: Medical Domain Hallucination Test for Large Language Modelscited by 157
  11. 2023Towards Accurate Differential Diagnosis with Large Language Modelscited by 116
  12. 2024Towards Conversational Diagnostic AIcited by 79
  13. 2023MAIRA-1: A specialised large multimodal model for radiology report generationcited by 74
  14. 2023Clinical Camel: An Open Expert-Level Medical Language Model with Dialogue-Based Knowledge Encodingcited by 71
  15. 2023MedAlign: A Clinician-Generated Dataset for Instruction Following with Electronic Medical Recordscited by 56
  16. 2021Worst of Both Worlds: Biases Compound in Pre-trained Vision-and-Language Modelscited by 52
  17. 2024Towards a clinically accessible radiology foundation model: open-access and lightweight, with automated evaluationcited by 51
  18. 2023ELIXR: Towards a general purpose X-ray artificial intelligence system through alignment of large language models and radiology vision encoderscited by 50
  19. 2022Revisiting Transformer-based Models for Long Document Classificationcited by 49
  20. 2024A Toolbox for Surfacing Health Equity Harms and Biases in Large Language Modelscited by 45
  21. 2022Towards Visual-Prompt Temporal Answering Grounding in Medical Instructional Videocited by 43
  22. 2023The Capability of Large Language Models to Measure Psychiatric Functioningcited by 26
  23. 2023Multimodal Composite Association Score: Measuring Gender Bias in Generative Multimodal Modelscited by 21
  24. 2023A Foundational Multimodal Vision Language AI Assistant for Human Pathologycited by 21
  25. 2024How well do LLMs cite relevant medical references? An evaluation framework and analysescited by 21
  26. 2024Crowdsourcing Dermatology Images with Google Search Ads: Creating a Real-World Skin Condition Datasetcited by 21
  27. 2023Multimodal Foundation Models Exploit Text to Make Medical Image Predictionscited by 20
  28. 2020Pathological Visual Question Answeringcited by 18
  29. 2021A real-time spatiotemporal AI model analyzes skill in open surgical videoscited by 11
  30. 2021Surgical data science for safe cholecystectomy: a protocol for segmentation of hepatocystic anatomy and assessment of the critical view of safetycited by 8

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 82; 2025: 86; 2026: 69.

237 papers.

Sort:

By area:

  1. 2025HealthBench: Evaluating Large Language Models Towards Improved Human Healthcited by 429
  2. 2024HuatuoGPT-o1, Towards Medical Complex Reasoning with LLMscited by 265
  3. 2025MedXpertQA: Benchmarking Expert-Level Medical Reasoning and Understandingcited by 224
  4. 2024NVILA: Efficient Frontier Visual Language Modelscited by 209
  5. 2024Two Tales of Persona in LLMs: A Survey of Role-Playing and Personalizationcited by 208
  6. 2024Poseidon: Efficient Foundation Models for PDEscited by 200
  7. 2024Are We Done with MMLU?cited by 194
  8. 2025MCP-Bench: Benchmarking Tool-Using LLM Agents with Complex Real-World Tasks via MCP Serverscited by 109
  9. 2024A Comprehensive Survey of Scientific Large Language Models and Their Applications in Scientific Discoverycited by 96
  10. 2024Advancing Multimodal Medical Capabilities of Geminicited by 90
  11. 2024Medical Graph RAG: Towards Safe Medical Large Language Model via Graph Retrieval-Augmented Generationcited by 90
  12. 2024MedTrinity-25M: A Large-scale Multimodal Dataset with Multigranular Annotations for Medicinecited by 81
38 more
  1. 2024MedCalc-Bench: Evaluating Large Language Models for Medical Calculationscited by 70
  2. 2024UltraMedical: Building Specialized Generalists in Biomedicinecited by 67
  3. 2024Improving Retrieval-Augmented Generation in Medicine with Iterative Follow-up Questionscited by 54
  4. 2024VILA-M3: Enhancing Vision-Language Models with Medical Expert Knowledgecited by 50
  5. 2024MedImageInsight: An Open-Source Embedding Model for General Domain Medical Imagingcited by 46
  6. 2024Rationale-Guided Retrieval Augmented Generation for Medical Question Answeringcited by 41
  7. 2025LLMs are Greedy Agents: Effects of RL Fine-tuning on Decision-Making Abilitiescited by 41
  8. 2025A Survey of Scientific Large Language Models: From Data Foundations to Agent Frontierscited by 37
  9. 2025Beyond Distillation: Pushing the Limits of Medical LLM Reasoning with Minimalist Rule-Based RLcited by 36
  10. 2024A Survey on Medical Large Language Models: Technology, Application, Trustworthiness, and Future Directionscited by 34
  11. 2025BIOMEDICA: An Open Biomedical Image-Caption Archive, Dataset, and Vision-Language Models Derived from Scientific Literaturecited by 34
  12. 2025MedFrameQA: A Multi-Image Medical VQA Benchmark for Clinical Reasoningcited by 34
  13. 2024Beyond Perplexity: Multi-dimensional Safety Evaluation of LLM Compressioncited by 33
  14. 2024MEDCO: Medical Education Copilots Based on A Multi-Agent Frameworkcited by 33
  15. 2025MicroVQA: A Multimodal Reasoning Benchmark for Microscopy-Based Scientific Researchcited by 32
  16. 2024Teach Multimodal LLMs to Comprehend Electrocardiographic Imagescited by 28
  17. 2024Medical Adaptation of Large Language and Vision-Language Models: Are We Making Progress?cited by 28
  18. 2024Superhuman performance of a large language model on the reasoning tasks of a physiciancited by 28
  19. 2024Towards a Personal Health Large Language Modelcited by 27
  20. 2024Finetuning Language Models to Emit Linguistic Expressions of Uncertaintycited by 26
  21. 2024DiffKV: Differentiated Memory Management for Large Language Models with Parallel KV Compactioncited by 26
  22. 2026HealthBench Professional: Evaluating Large Language Models on Real Clinician Chatscited by 26
  23. 2024A Benchmark for Long-Form Medical Question Answeringcited by 25
  24. 2025RadVLM: A Multitask Conversational Vision-Language Model for Radiologycited by 25
  25. 2024From Medprompt to o1: Exploration of Run-Time Strategies for Medical Challenge Problems and Beyondcited by 24
  26. 2025MEDDxAgent: A Unified Modular Agent Framework for Explainable Automatic Differential Diagnosiscited by 23
  27. 2025FEABench: Evaluating Language Models on Multiphysics Reasoning Abilitycited by 22
  28. 2024Biomedical Visual Instruction Tuning with Clinician Preference Alignmentcited by 21
  29. 2024MedBench: A Comprehensive, Standardized, and Reliable Benchmarking System for Evaluating Chinese Medical Large Language Modelscited by 21
  30. 2025Advancing Conversational Diagnostic AI with Multimodal Reasoningcited by 20
  31. 2025Ask Patients with Patience: Enabling LLMs for Human-Centric Medical Dialogue with Grounded Reasoningcited by 19
  32. 2024GP-VLS: A general-purpose vision language model for surgerycited by 17
  33. 2025From Problem-Solving to Teaching Problem-Solving: Aligning LLMs with Pedagogy using Reinforcement Learningcited by 17
  34. 2025Citrus: Leveraging Expert Cognitive Pathways in a Medical Language Model for Advanced Medical Decision Supportcited by 16
  35. 2025Mitigating Parameter Interference in Model Merging via Sharpness-Aware Fine-Tuningcited by 16
  36. 2026MedMO: Grounding and Understanding Multimodal Large Language Model for Medical Imagescited by 15
  37. 2025Med-CMR: A Fine-Grained Benchmark Integrating Visual Evidence and Clinical Logic for Medical Complex Multimodal Reasoningcited by 14
  38. 2024Pattern Recognition or Medical Knowledge? The Problem with Multiple-Choice Questions in Medicinecited by 13

50 most cited of 237 All 237 (JSON)