mlsanityData to 14 Oct 2026

From Text to Multimodality: Exploring the Evolution and Impact of Large Language Models in Medical Practice

arXiv 2410.01812 ↗First posted 14 Sep 2024cs.CY cs.AI cs.CLcited by 8

Not on the map. Why?

Abstract (arXiv)

Large Language Models (LLMs) have rapidly evolved from text-based systems to multimodal platforms, significantly impacting various sectors including healthcare. This comprehensive review explores the progression of LLMs to Multimodal Large Language Models (MLLMs) and their growing influence in medical practice. We examine the current landscape of MLLMs in healthcare, analyzing their applications across clinical decision support, medical imaging, patient engagement, and research. The review highlights the unique capabilities of MLLMs in integrating diverse data types, such as text, images, and audio, to provide more comprehensive insights into patient health. We also address the challenges facing MLLM implementation, including data limitations, technical hurdles, and ethical considerations. By identifying key research gaps, this paper aims to guide future investigations in areas such as dataset development, modality alignment methods, and the establishment of ethical guidelines. As MLLMs continue to shape the future of healthcare, understanding their potential and limitations is crucial for their responsible and effective integration into medical practice.

Key facts. Cited by 8 papers in the map (data through 14 Oct 2026). Builds on 59 of its 112 references that are in the map. First posted on arXiv 14 Sep 2024.

Builds on

59 of 112 references.

References · 59

  1. 2017Attention Is All You Needcited by 60,361
  2. 2023GPT-4 Technical Reportcited by 18,396
  3. 2022Large Language Models Encode Clinical Knowledgecited by 1,830
  4. 2023LLaVA-Med: Training a Large Language-and-Vision Assistant for Biomedicine in One Daycited by 1,305
  5. 2023mPLUG-Owl2: Revolutionizing Multi-modal Large Language Model with Modality Collaborationcited by 547
  6. 2023BiomedCLIP: a multimodal biomedical foundation model pretrained from fifteen million scientific image-text pairscited by 520
  7. 2023Med-Flamingo: a Multimodal Medical Few-shot Learnercited by 392
  8. 2023Towards Generalist Biomedical AIcited by 289
  9. 2024MM-LLMs: Recent Advances in MultiModal Large Language Modelscited by 278
  10. 2024AgentClinic: a multimodal agent benchmark to evaluate AI in simulated clinical environmentscited by 231
  11. 2024OmniMedVQA: A New Large-Scale Comprehensive Evaluation Benchmark for Medical LVLMcited by 184
  12. 2023A Survey of Large Language Models for Healthcare: from Data, Technology, and Applications to Accountability and Ethicscited by 163
38 more
  1. 2023Med-HALT: Medical Domain Hallucination Test for Large Language Modelscited by 157
  2. 2023X-LLM: Bootstrapping Advanced Large Language Models by Treating Multi-Modalities as Foreign Languagescited by 128
  3. 2024Vision-Language Models for Medical Report Generation and Visual Question Answering: A Reviewcited by 125
  4. 2023Almanac: Retrieval-Augmented Language Models for Clinical Medicinecited by 123
  5. 2023Towards Accurate Differential Diagnosis with Large Language Modelscited by 116
  6. 2024Towards Building Multilingual Language Model for Medicinecited by 92
  7. 2023PMC-LLaMA: Towards Building Open-source Language Models for Medicinecited by 88
  8. 2023ClinicalGPT: Large Language Models Finetuned with Diverse Medical Data and Comprehensive Evaluationcited by 88
  9. 2023Better to Ask in English: Cross-Lingual Evaluation of Large Language Models for Healthcare Queriescited by 82
  10. 2021MMBERT: Multimodal BERT Pretraining for Improved Medical VQAcited by 68
  11. 2022Multimodal learning with graphscited by 54
  12. 2024Large Language Models and Cognitive Science: A Comprehensive Review of Similarities, Differences, and Challengescited by 49
  13. 2024Multimodal Healthcare AI: Identifying and Designing Clinically Relevant Vision-Language Applications for Radiologycited by 45
  14. 2023Radiology-GPT: A Large Language Model for Radiologycited by 44
  15. 2024MedSafetyBench: Evaluating and Improving the Medical Safety of Large Language Modelscited by 44
  16. 2023ChatGPT for Shaping the Future of Dentistry: The Potential of Multi-Modal Large Language Modelcited by 40
  17. 2024Quantifying and Mitigating Unimodal Biases in Multimodal Large Language Models: A Causal Perspectivecited by 39
  18. 2024MedVersa: A Generalist Foundation Model for Medical Image Interpretationcited by 38
  19. 2023Language Models are Few-shot Learners for Prognostic Predictioncited by 37
  20. 2023Taiyi: A Bilingual Fine-Tuned Large Language Model for Diverse Biomedical Taskscited by 36
  21. 2024Addressing cognitive bias in medical language modelscited by 36
  22. 2024GSCo: Towards Generalizable AI in Medicine via Generalist-Specialist Collaborationcited by 34
  23. 2024JMLR: Joint Medical LLM and Retrieval Training for Enhancing Reasoning and Professional Question Answering Capabilitycited by 32
  24. 2024A Review of Multi-Modal Large Language and Vision Modelscited by 30
  25. 2024On Large Visual Language Models for Medical Imaging Analysis: An Empirical Studycited by 28
  26. 2024Gemini Goes to Med School: Exploring the Capabilities of Multimodal Large Language Models on Medical Challenge Problems & Hallucinationscited by 27
  27. 2024Securing Large Language Models: Addressing Bias, Misinformation, and Prompt Attackscited by 26
  28. 2024REALM: RAG-Driven Enhancement of Multimodal Electronic Health Records Analysis via Large Language Modelscited by 23
  29. 2024Prompting Large Language Models for Zero-Shot Clinical Prediction with Structured Longitudinal Electronic Health Record Datacited by 22
  30. 2024MedSumm: A Multimodal Approach to Summarizing Code-Mixed Hindi-English Clinical Queriescited by 19
  31. 2024Unbridled Icarus: A Survey of the Potential Perils of Image Inputs in Multimodal Large Language Model Securitycited by 19
  32. 2024The Sound of Healthcare: Improving Medical Transcription ASR Accuracy with Large Language Modelscited by 18
  33. 2024Health-LLM: Personalized Retrieval-Augmented Disease Prediction Systemcited by 17
  34. 2024A Comprehensive Survey on Evaluating Large Language Model Applications in the Medical Industrycited by 17
  35. 2024Hippocrates: An Open-Source Framework for Advancing Large Language Models in Healthcarecited by 15
  36. 2024MISS: A Generative Pretraining and Finetuning Approach for Med-VQAcited by 14
  37. 2023Interpretable Medical Diagnostics with Structured Data Extraction by Large Language Modelscited by 13
  38. 2024Large Language Models in the Clinic: A Comprehensive Benchmarkcited by 13

50 most cited of 59 All 59 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 2; 2025: 4; 2026: 2.

8 papers.

  1. 2024Large Language Model Benchmarks in Medical Taskscited by 17
  2. 2025Mitigating Visual Knowledge Forgetting in MLLM Instruction-tuning via Modality-decoupled Gradient Descentcited by 13
  3. 2025Speeding up Speculative Decoding via Sequential Approximate Verificationcited by 11
  4. 2025Retrieval-augmented in-context learning for multimodal large language models in disease classificationcited by 5
  5. 2024The Cat and Mouse Game: The Ongoing Arms Race Between Diffusion Models and Detection Methodscited by 4
  6. 2025SwasthLLM: a Unified Cross-Lingual, Multi-Task, and Meta-Learning Zero-Shot Framework for Medical Diagnosis Using Contrastive Representationscited by 0
  7. 2026EVLA: An Electro-Aware Multimodal Assistant for Physically-Grounded Driving Reasoning and Controlcited by 0
  8. 2026Beyond Skepticism: Evaluating LLMs Pedagogical Intent Reasoning with the Adaptive Pedagogical Vigilance Frameworkcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024A Comprehensive Survey of Large Language Models and Multimodal Large Language Models in Medicine
  2. 2025Multimodal Large Language Models for Medicine: A Comprehensive Survey
  3. 2025Exploring Large Language Models in Healthcare: Insights into Corpora Sources, Customization Strategies, and Evaluation Metrics
  4. 2026A primer on evaluation methods for large language models in healthcare
  5. 2025BRIDGE: Benchmarking Large Language Models for Understanding Real-world Clinical Practice Text
  6. 2024Surveying the MLLM Landscape: A Meta-Review of Current Surveys
  7. 2025Vision Language Models in Medicine
  8. 2025Reviewing Clinical Knowledge in Medical Large Language Models: Training and Beyond