mlsanityData to 14 Oct 2026

A Comprehensive Survey of Large Language Models and Multimodal Large Language Models in Medicine

arXiv 2405.08603 ↗First posted 14 May 2024cs.CLcited by 45

On the map: Multimodal LLMs / vision-language models → Medical vision-language models

Abstract (arXiv)

Since the release of ChatGPT and GPT-4, large language models (LLMs) and multimodal large language models (MLLMs) have attracted widespread attention for their exceptional capabilities in understanding, reasoning, and generation, introducing transformative paradigms for integrating artificial intelligence into medicine. This survey provides a comprehensive overview of the development, principles, application scenarios, challenges, and future directions of LLMs and MLLMs in medicine. Specifically, it begins by examining the paradigm shift, tracing the transition from traditional models to LLMs and MLLMs, and highlighting the unique advantages of these LLMs and MLLMs in medical applications. Next, the survey reviews existing medical LLMs and MLLMs, providing detailed guidance on their construction and evaluation in a clear and systematic manner. Subsequently, to underscore the substantial value of LLMs and MLLMs in healthcare, the survey explores five promising applications in the field. Finally, the survey addresses the challenges confronting medical LLMs and MLLMs and proposes practical strategies and future directions for their integration into medicine. In summary, this survey offers a comprehensive analysis of the technical methodologies and practical clinical applications of medical LLMs and MLLMs, with the goal of bridging the gap between these advanced technologies and clinical practice, thereby fostering the evolution of the next generation of intelligent healthcare systems.

Key facts. Cited by 45 papers in the map (data through 14 Oct 2026). Placed in Medical vision-language models, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 190 of its 313 references that are in the map. First posted on arXiv 14 May 2024.

Builds on

190 of 313 references.

Closest ancestors · 7

Same direction, newest first

  1. 2024M3D: Advancing 3D Medical Image Analysis with Multi-Modal Large Language Models31 Mar
  2. 2023Qilin-Med-VL: Towards Chinese Large Vision-Language Model for General Healthcare27 Oct
  3. 2023OphGLM: Training an Ophthalmology Large Language-and-Vision Assistant based on Instructions and Dialogue21 Jun
  4. 2023XrayGPT: Chest Radiographs Summarization using Medical Vision-Language Models13 Jun
  5. 2023LLaVA-Med: Training a Large Language-and-Vision Assistant for Biomedicine in One Day1 Jun
  6. 2023PathAsst: A Generative Foundation AI Assistant Towards Artificial General Intelligence of Pathology24 May
  7. 2023MedBLIP: Bootstrapping Language-Image Pre-training from 3D Medical Images and Texts18 May

Other references · 183

  1. 2017Attention Is All You Needcited by 60,361
  2. 2015Deep Residual Learning for Image Recognitioncited by 52,096
  3. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  4. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  5. 2020Language Models are Few-Shot Learnerscited by 32,419
  6. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 30,186
  7. 2023GPT-4 Technical Reportcited by 18,396
  8. 2022Training language models to follow instructions with human feedbackcited by 15,971
  9. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,599
  10. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 14,132
  11. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,372
  12. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 13,028
38 more
  1. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,626
  2. 2023Segment Anythingcited by 8,234
  3. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 8,065
  4. 2023DINOv2: Learning Robust Visual Features without Supervisioncited by 7,974
  5. 2023Visual Instruction Tuningcited by 7,869
  6. 2020Scaling Laws for Neural Language Modelscited by 6,586
  7. 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,137
  8. 2022Self-Consistency Improves Chain of Thought Reasoning in Language Modelscited by 5,708
  9. 2019BERTScore: Evaluating Text Generation with BERTcited by 5,512
  10. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,326
  11. 2023Mamba: Linear-Time Sequence Modeling with Selective State Spacescited by 4,718
  12. 2023Improved Baselines with Visual Instruction Tuningcited by 4,418
  13. 2022Flamingo: a Visual Language Model for Few-Shot Learningcited by 4,386
  14. 2022Large Language Models are Zero-Shot Reasonerscited by 4,374
  15. 2019Parameter-Efficient Transfer Learning for NLPcited by 4,236
  16. 2022Robust Speech Recognition via Large-Scale Weak Supervisioncited by 4,057
  17. 2021Prefix-Tuning: Optimizing Continuous Prompts for Generationcited by 3,915
  18. 2021The Power of Scale for Parameter-Efficient Prompt Tuningcited by 3,795
  19. 2021Finetuned Language Models Are Zero-Shot Learnerscited by 3,780
  20. 2023Self-Refine: Iterative Refinement with Self-Feedbackcited by 3,442
  21. 2023Qwen Technical Reportcited by 3,344
  22. 2023Grounding DINO: Marrying DINO with Grounded Pre-Training for Open-Set Object Detectioncited by 3,332
  23. 2023Tree of Thoughts: Deliberate Problem Solving with Large Language Modelscited by 3,211
  24. 2023Mistral 7Bcited by 3,164
  25. 2022Constitutional AI: Harmlessness from AI Feedbackcited by 3,050
  26. 2014Sequence to Sequence Learning with Neural Networkscited by 2,986
  27. 2023InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuningcited by 2,876
  28. 2019XLNet: Generalized Autoregressive Pretraining for Language Understandingcited by 2,811
  29. 2019ALBERT: A Lite BERT for Self-supervised Learning of Language Representationscited by 2,790
  30. 2022Self-Instruct: Aligning Language Models with Self-Generated Instructionscited by 2,724
  31. 2022Locating and Editing Factual Associations in GPTcited by 2,691
  32. 2020Learning to summarize from human feedbackcited by 2,588
  33. 2021Pre-train, Prompt, and Predict: A Systematic Survey of Prompting Methods in Natural Language Processingcited by 2,234
  34. 2020Extracting Training Data from Large Language Modelscited by 2,189
  35. 2020DeBERTa: Decoding-enhanced BERT with Disentangled Attentioncited by 2,181
  36. 2022Survey of Hallucination in Natural Language Generationcited by 2,075
  37. 2014CIDEr: Consensus-based Image Description Evaluationcited by 2,046
  38. 2019BioBERT: a pre-trained biomedical language representation model for biomedical text miningcited by 1,947

50 most cited of 183 All 183 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 13; 2025: 22; 2026: 10.

45 papers.

Sort:

By area:

  1. 2024SkySenseGPT: A Fine-Grained Instruction Tuning Dataset and Model for Remote Sensing Vision-Language Understandingcited by 104
  2. 2024MultiTrust: A Comprehensive Benchmark Towards Trustworthy Multimodal Large Language Modelscited by 53
  3. 2024Explainable and Interpretable Multimodal Large Language Models: A Comprehensive Surveycited by 52
  4. 2024ErrorRadar: Benchmarking Complex Mathematical Reasoning of Multimodal Large Language Models Via Error Detectioncited by 37
  5. 2025Membership Inference Attacks Against Vision-Language Modelscited by 29
  6. 2024UrBench: A Comprehensive Benchmark for Evaluating Large Multimodal Models in Multi-View Urban Scenarioscited by 28
  7. 2024Towards a Personal Health Large Language Modelcited by 27
  8. 2024[CLS] Token Tells Everything Needed for Training-free Efficient MLLMscited by 26
  9. 2024From Specific-MLLMs to Omni-MLLMs: A Survey on MLLMs Aligned with Multi-modalitiescited by 22
  10. 2024MedAide: Information Fusion and Anatomy of Medical Intents via LLM-based Agent Collaborationcited by 16
  11. 2025Multimodal Large Language Models for Medicine: A Comprehensive Surveycited by 16
  12. 2024Simulated patient systems powered by large language model-based AI agents offer potential for transforming medical educationcited by 13
33 more
  1. 2025mCLM: A Modular Chemical Language Model that Generates Functional and Makeable Moleculescited by 8
  2. 2025Odysseus: Jailbreaking Commercial Multimodal LLM-integrated Systems via Dual Steganographycited by 8
  3. 2024LMOD: A Large Multimodal Ophthalmology Dataset and Benchmark for Large Vision-Language Modelscited by 7
  4. 2024MINER: Mining the Underlying Pattern of Modality-Specific Neurons in Multimodal Large Language Modelscited by 7
  5. 2025Applications of Large Models in Medicinecited by 6
  6. 2025UniVRSE: Unified Vision-conditioned Response Semantic Entropy for Hallucination Detection in Medical Vision-Language Modelscited by 5
  7. 2025EVADE-Bench: Multimodal Benchmark for Evaluating and Enhancing Evasive Content Detectioncited by 5
  8. 2025Large Language Models for Bioinformaticscited by 3
  9. 2025Look & Mark: Leveraging Radiologist Eye Fixations and Bounding boxes in Multimodal Large Language Models for Chest X-ray Report Generationcited by 3
  10. 2025Large Language models for Time Series Analysis: Techniques, Applications, and Challengescited by 3
  11. 2026The Judgment-Consequence Gap: LLM Moral Reasoning in Healthcare Decisionscited by 3
  12. 2025Who Gets the Kidney? Human-AI Alignment, Indecision, and Moral Valuescited by 2
  13. 2025MedMKEB: A Comprehensive Knowledge Editing Benchmark for Medical Multimodal Large Language Modelscited by 2
  14. 2025Direct Token Optimization: A Self-contained Approach to Large Language Model Unlearningcited by 2
  15. 2025Perceive and Calibrate: Analyzing and Enhancing Robustness of Medical Multi-Modal Large Language Modelscited by 2
  16. 2025Fairness in Multi-modal Medical Diagnosis with Demonstration Selectioncited by 1
  17. 2026AnatomiX, an Anatomy-Aware Grounded Multimodal Large Language Model for Chest X-Ray Interpretationcited by 1
  18. 2026Measuring Cross-Modal Synergy: A Benchmark for VLM Explainabilitycited by 1
  19. 2026Document Classification Pattern Recognition via Information Fusion: A Systematic Review of Multimodal and Multiview Representation Approachescited by 1
  20. 2026BreastGPT: A Multimodal Large Language Model for the Full Spectrum of Breast Cancer Clinical Routinecited by 1
  21. 2024Reasoning-Aware Query-Focused Summarization over Multi-Table Datacited by 0
  22. 2025BiomechGPT: Extending Motion-Language Models to Clinical Motion Understandingcited by 0
  23. 2025CKD-EHR:Clinical Knowledge Distillation for Electronic Health Recordscited by 0
  24. 2025OR-VSKC: Resolving Visual-Semantic Knowledge Conflicts in Operating Rooms with Synthetic Data-Guided Alignmentcited by 0
  25. 2025RadEyeVideo: Enhancing general-domain Large Vision Language Model for chest X-ray analysis with video representations of eye gazecited by 0
  26. 2025VL-MedGuide: A Visual-Linguistic Large Model for Intelligent and Explainable Skin Disease Auxiliary Diagnosiscited by 0
  27. 2025HA-RAG: Hotness-Aware RAG Acceleration via Mixed Precision and Data Placementcited by 0
  28. 2025MEEA: Mere Exposure Effect-Driven Confrontational Optimization for LLM Jailbreakingcited by 0
  29. 2026MUNIChus: Multilingual News Image Captioning Benchmarkcited by 0
  30. 2026Instruction-Free Tuning of Large Vision Language Models for Medical Instruction Followingcited by 0
  31. 2026When Minor Edits Matter: LLM-Driven Prompt Attack for Medical VLM Robustness in Ultrasoundcited by 0
  32. 2026Deterministic Hallucination Detection in Medical VQA via Confidence-Evidence Bayesian Gaincited by 0
  33. 2026Evo-PI: Aligning Medical Reasoning via Evolving Principle-Guided Supervisioncited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024From Text to Multimodality: Exploring the Evolution and Impact of Large Language Models in Medical Practice
  2. 2023Large Language Models Illuminate a Progressive Pathway to Artificial Healthcare Assistant: A Review
  3. 2024A Survey for Large Language Models in Biomedicine
  4. 2024Entry-level guide to the use of large language models for medical research
  5. 2025A Survey of LLM-based Agents in Medicine: How far are we from Baymax?
  6. 2025Performance of Large Language Models in Supporting Medical Diagnosis and Treatment
  7. 2026A primer on evaluation methods for large language models in healthcare
  8. 2023From Beginner to Expert: Modeling Medical Knowledge into General LLMs