mlsanityData to 7 Oct 2026

MedViLaM: A multimodal large language model with advanced generalizability and explainability for medical data understanding and generation

arXiv 2409.19684 ↗First posted 29 Sep 2024cs.CVcited by 17

On the map: Multimodal LLMs / vision-language models → Medical vision-language models

Abstract (arXiv)

Medicine is inherently multimodal and multitask, with diverse data modalities spanning text, imaging. However, most models in medical field are unimodal single tasks and lack good generalizability and explainability. In this study, we introduce MedViLaM, a unified vision-language model towards a generalist model for medical data that can flexibly encode and interpret various forms of medical data, including clinical language and imaging, all using the same set of model weights. To facilitate the creation of such multi-task model, we have curated MultiMedBench, a comprehensive pretaining dataset and benchmark consisting of several distinct tasks, i.e., continuous question-answering, multi-label disease classification, disease localization, generation and summarization of radiology reports. MedViLaM demonstrates strong performance across all MultiMedBench tasks, frequently outpacing other generalist models by a significant margin. Additionally, we present instances of zero-shot generalization to new medical concepts and tasks, effective transfer learning across different tasks, and the emergence of zero-shot medical reasoning.

Key facts. Cited by 17 papers in the map (data through 7 Oct 2026). Placed in Medical vision-language models, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 23 of its 45 references that are in the map. First posted on arXiv 29 Sep 2024.

Builds on

23 of 45 references.

References · 23

  1. 2018YOLOv3: An Incremental Improvementcited by 2,652
  2. 2019CheXpert: A Large Chest Radiograph Dataset with Uncertainty Labels and Expert Comparisoncited by 1,410
  3. 2017ChestX-ray8: Hospital-scale Chest X-ray Database and Benchmarks on Weakly-Supervised Classification and Localization of Common Thorax Diseasescited by 969
  4. 2016EndoNet: A Deep Architecture for Recognition Tasks on Laparoscopic Videoscited by 366
  5. 2022Making the Most of Text Semantics to Improve Biomedical Vision--Language Processingcited by 336
  6. 2019PadChest: A large chest x-ray image dataset with multi-label annotated reportscited by 308
  7. 2020COVID-Net: A Tailored Deep Convolutional Neural Network Design for Detection of COVID-19 Cases from Chest X-Ray Imagescited by 264
  8. 2020VinDr-CXR: An open dataset of chest X-rays with radiologist's annotationscited by 236
  9. 2020Can AI help in screening Viral and COVID-19 pneumonia?cited by 200
  10. 2016Validation, comparison, and combination of algorithms for automatic detection of pulmonary nodules in computed tomography images: the LUNA16 challengecited by 198
  11. 2016Skin Lesion Analysis toward Melanoma Detection: A Challenge at the International Symposium on Biomedical Imaging (ISBI) 2016, hosted by the International Skin Imaging Collaboration (ISIC)cited by 197
  12. 2020COVID-19 Image Data Collectioncited by 100
11 more
  1. 2022VinDr-Mammo: A large-scale benchmark dataset for computer-aided diagnosis in full-field digital mammographycited by 83
  2. 2020BIMCV COVID-19+: a large annotated dataset of RX and CT images from COVID-19 patientscited by 81
  3. 2022Video Polyp Segmentation: A Deep Learning Perspectivecited by 81
  4. 2020ISINet: An Instance-Based Approach for Surgical Instrument Segmentationcited by 67
  5. 2020Coswara -- A Database of Breathing, Cough, and Voice Sounds for COVID-19 Diagnosiscited by 58
  6. 2021Visual Grounding with Transformerscited by 42
  7. 2021A Structure-Aware Relation Network for Thoracic Diseases Detection and Segmentationcited by 33
  8. 2023Segmentation and Vascular Vectorization for Coronary Artery by Geometry-based Cascaded Neural Networkcited by 21
  9. 2022Contrastive Transformer-based Multiple Instance Learning for Weakly Supervised Polyp Frame Detectioncited by 20
  10. 2021VinDr-SpineXR: A deep learning framework for spinal lesions detection and classification from radiographscited by 19
  11. 2021DICOM Imaging Router: An Open Deep Learning Framework for Classification of Body Parts from DICOM X-ray Scanscited by 11

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 1; 2025: 4; 2026: 12.

17 papers.

Sort:

By area:

  1. 2025A Comprehensive Survey on the Trustworthiness of Large Language Models in Healthcarecited by 27
  2. 2025Multimodal Large Language Models for Medicine: A Comprehensive Surveycited by 16
  3. 2024A foundation model for generalizable disease diagnosis in chest X-ray imagescited by 12
  4. 2026The Model Knows Which Tokens Matter: Automatic Token Selection via Noise Gatingcited by 11
  5. 2026Efficient Chest X-ray Representation Learning via Semantic-Partitioned Contrastive Learningcited by 10
  6. 2026Beyond Surrogate Gradients: Fully Differentiable Token Pruning for Vision-Language Modelscited by 7
  7. 2026Multimodal Model for Computational Pathology:Representation Learning and Image Compressioncited by 6
  8. 2026XrayClaw: Cooperative-Competitive Multi-Agent Alignment for Trustworthy Chest X-ray Diagnosiscited by 5
  9. 2026A unified multi-task framework enables interpretable chest radiograph analysiscited by 5
  10. 2026Stepwise Token Selection for Efficient Multimodal Large Language Modelscited by 5
  11. 2026Learnable Token Sparsification for Efficient Gigapixel Whole Slide Image Reasoningcited by 4
  12. 2025Point, Detect, Count: Multi-Task Medical Image Understanding with Instruction-Tuned Vision-Language Modelscited by 2
5 more
  1. 2025Perceive and Calibrate: Analyzing and Enhancing Robustness of Medical Multi-Modal Large Language Modelscited by 2
  2. 2026PathSelect: Sequential Token Selection for Whole Slide Pathologycited by 2
  3. 2026DiffPrune: differentiable information throttling for token pruning in vision-language modelscited by 1
  4. 2026From Snapshots to Symphonies: The Evolution of Protein Prediction from Static Structures to Generative Dynamics and Multimodal Interactionscited by 0
  5. 2026StepPrune: Adaptive Sequential Visual Token Selection across Multimodal Large Language Modelscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Lingshu: A Generalist Foundation Model for Unified Multimodal Medical Understanding and Reasoning
  2. 2024PeFoMed: Parameter Efficient Fine-tuning of Multimodal Large Language Models for Medical Imaging
  3. 2026MedMO: Grounding and Understanding Multimodal Large Language Model for Medical Images
  4. 2026MedUAG: Unified Understanding and Generation for Medical Multimodal Models
  5. 2025MedMKEB: A Comprehensive Knowledge Editing Benchmark for Medical Multimodal Large Language Models
  6. 2025SMMILE: An Expert-Driven Benchmark for Multimodal Medical In-Context Learning
  7. 2025MedGEN-Bench: A Contextually Entangled Benchmark for Open-ended Multimodal Medical Generation
  8. 2025MedAtlas: Evaluating LLMs for Multi-Round, Multi-Task Medical Reasoning Across Diverse Imaging Modalities and Clinical Text