mlsanityData to 14 Oct 2026

BiomedGPT: A Generalist Vision-Language Foundation Model for Diverse Biomedical Tasks

arXiv 2305.17100 ↗First posted 26 May 2023cs.CL cs.AIcited by 201

On the map: Multimodal LLMs / vision-language models → Medical vision-language models

Abstract (arXiv)

Traditional biomedical artificial intelligence (AI) models, designed for specific tasks or modalities, often exhibit limited flexibility in real-world deployment and struggle to utilize holistic information. Generalist AI holds the potential to address these limitations due to its versatility in interpreting different data types and generating tailored outputs for diverse needs. However, existing biomedical generalist AI solutions are typically heavyweight and closed source to researchers, practitioners, and patients. Here, we propose BiomedGPT, the first open-source and lightweight vision-language foundation model, designed as a generalist capable of performing various biomedical tasks. BiomedGPT achieved state-of-the-art results in 16 out of 25 experiments while maintaining a computing-friendly model scale. We also conducted human evaluations to assess the capabilities of BiomedGPT in radiology visual question answering, report generation, and summarization. BiomedGPT exhibits robust prediction ability with a low error rate of 3.8% in question answering, satisfactory performance with an error rate of 8.3% in writing complex radiology reports, and competitive summarization ability with a nearly equivalent preference score to human experts. Our method demonstrates that effective training with diverse data can lead to more practical biomedical AI for improving diagnosis and workflow efficiency.

Key facts. Cited by 201 papers in the map (data through 14 Oct 2026). Placed in Medical vision-language models, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 40 of its 77 references that are in the map. First posted on arXiv 26 May 2023.

Builds on

40 of 77 references.

Closest ancestors · 1

Same direction, newest first

  1. 2022Self-supervised vision-language pretraining for Medical visual question answering24 Nov

Other references · 39

  1. 2017Attention Is All You Needcited by 60,361
  2. 2015Deep Residual Learning for Image Recognitioncited by 52,096
  3. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  4. 2020Language Models are Few-Shot Learnerscited by 32,419
  5. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 30,186
  6. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 14,132
  7. 2016Layer Normalizationcited by 5,669
  8. 2021Finetuned Language Models Are Zero-Shot Learnerscited by 3,780
  9. 2020Taming Transformers for High-Resolution Image Synthesiscited by 3,253
  10. 2014Sequence to Sequence Learning with Neural Networkscited by 2,986
  11. 2023PaLM-E: An Embodied Multimodal Language Modelcited by 2,071
  12. 2017Mixed Precision Trainingcited by 1,156
27 more
  1. 2022A Generalist Agentcited by 808
  2. 2021MedMNIST v2 -- A large-scale lightweight benchmark for 2D and 3D biomedical image classificationcited by 690
  3. 2020It's Not Just Size That Matters: Small Language Models Are Also Few-Shot Learnerscited by 669
  4. 2021SimVLM: Simple Visual Language Model Pretraining with Weak Supervisioncited by 647
  5. 2021CoAtNet: Marrying Convolution and Attention for All Data Sizescited by 569
  6. 2022BioGPT: Generative Pre-trained Transformer for Biomedical Text Generation and Miningcited by 538
  7. 2023BiomedCLIP: a multimodal biomedical foundation model pretrained from fifteen million scientific image-text pairscited by 520
  8. 2021SLAKE: A Semantically-Labeled Knowledge-Enhanced Dataset for Medical Visual Question Answeringcited by 449
  9. 2019Transfusion: Understanding Transfer Learning for Medical Imagingcited by 368
  10. 2020Generating Radiology Reports via Memory-driven Transformercited by 368
  11. 2021Pix2seq: A Language Modeling Framework for Object Detectioncited by 349
  12. 2023PMC-VQA: Visual Instruction Tuning for Medical Visual Question Answeringcited by 338
  13. 2017On the Automatic Generation of Medical Imaging Reportscited by 216
  14. 2020Rethinking Positional Encoding in Language Pre-trainingcited by 197
  15. 2021Exploring and Distilling Posterior and Prior Knowledge for Radiology Report Generationcited by 162
  16. 2018Lessons from Natural Language Inference in the Clinical Domaincited by 151
  17. 2021SciFive: a text-to-text transformer model for biomedical literaturecited by 99
  18. 2023MedViT: A Robust Vision Transformer for Generalized Medical Image Classificationcited by 89
  19. 2021E2E-VLP: End-to-End Vision-Language Pre-training Enhanced by Visual Learningcited by 85
  20. 2021NormFormer: Improved Transformer Pretraining with Extra Normalizationcited by 83
  21. 2022BioBART: Pretraining and Evaluation of A Biomedical Generative Language Modelcited by 83
  22. 2022Improving the Factual Correctness of Radiology Report Generation with Semantic Rewardscited by 83
  23. 2023Open-Ended Medical Visual Question Answering Through Prefix Tuning of Language Modelscited by 58
  24. 2021Clinical Outcome Prediction from Admission Notes using Self-Supervised Knowledge Integrationcited by 44
  25. 2023Multimodal Image-Text Matching Improves Retrieval-based Chest X-Ray Report Generationcited by 37
  26. 2019A Survey on Biomedical Image Captioningcited by 30
  27. 2023RadAdapt: Radiology Report Summarization via Lightweight Domain Adaptation of Large Language Modelscited by 24

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 15; 2024: 47; 2025: 99; 2026: 40.

201 papers.

Sort:

By area:

  1. 2024MLLM-as-a-Judge: Assessing Multimodal LLM-as-a-Judge with Vision-Language Benchmarkcited by 372
  2. 2023Towards Generalist Biomedical AIcited by 289
  3. 2025Lingshu: A Generalist Foundation Model for Unified Multimodal Medical Understanding and Reasoningcited by 230
  4. 2025MedVLM-R1: Incentivizing Medical Reasoning Capability of Vision-Language Models (VLMs) via Reinforcement Learningcited by 173
  5. 2025Baichuan-Omni-1.5 Technical Reportcited by 111
  6. 2024Advancing Multimodal Medical Capabilities of Geminicited by 90
  7. 2024Foundation Model for Advancing Healthcare: Challenges, Opportunities, and Future Directionscited by 77
  8. 2024MME-Survey: A Comprehensive Survey on Evaluation of Multimodal LLMscited by 72
  9. 2025Sparse Autoencoders Learn Monosemantic Features in Vision-Language Modelscited by 71
  10. 2024A Comprehensive Review of Multimodal Large Language Models: Performance and Challenges Across Different Taskscited by 57
  11. 2024MedImageInsight: An Open-Source Embedding Model for General Domain Medical Imagingcited by 46
  12. 2023SAMAug: Point Prompt Augmentation for Segment Anything Modelcited by 38
38 more
  1. 2023Multimodal ChatGPT for Medical Applications: an Experimental Study of GPT-4Vcited by 38
  2. 2025QoQ-Med: Building Multimodal Clinical Foundation Models with Domain-Aware GRPO Trainingcited by 38
  3. 2025EndoChat: Grounded Multimodal Large Language Model for Endoscopic Surgerycited by 37
  4. 2025RadGPT: Constructing 3D Image-Text Tumor Datasetscited by 36
  5. 2024CPath-Omni: A Unified Multimodal Foundation Model for Patch and Whole Slide Image Analysis in Computational Pathologycited by 34
  6. 2023Artificial General Intelligence for Medical Imaging Analysiscited by 32
  7. 2023Foundation Metrics for Evaluating Effectiveness of Healthcare Conversations Powered by Generative AIcited by 32
  8. 2025Limitations of Large Language Models in Clinical Problem-Solving Arising from Inflexible Reasoningcited by 31
  9. 2024MultiMedEval: A Benchmark and a Toolkit for Evaluating Medical Vision-Language Modelscited by 29
  10. 2025Derm1M: A Million-scale Vision-Language Dataset Aligned with Clinical Ontology Knowledge for Dermatologycited by 28
  11. 2024Opportunities and Challenges of Large Language Models for Low-Resource Languages in Humanities Researchcited by 27
  12. 2024BrainWave: A Brain Signal Foundation Model for Clinical Applicationscited by 26
  13. 2024Efficient Continual Pre-training by Mitigating the Stability Gapcited by 25
  14. 2024ReXrank: A Public Leaderboard for AI-Powered Radiology Report Generationcited by 25
  15. 2023Medical Vision Language Pretraining: A surveycited by 24
  16. 2024Interpretable Bilingual Multimodal Large Language Model for Diverse Biomedical Taskscited by 24
  17. 2024GMAI-VL & GMAI-VL-5.5M: A Large Vision-Language Model and A Comprehensive Multimodal Dataset Towards General Medical AIcited by 24
  18. 2025Triad: Vision Foundation Model for 3D Magnetic Resonance Imagingcited by 24
  19. 2024BiMediX2: Bio-Medical EXpert LMM for Diverse Medical Modalitiescited by 20
  20. 2025A Modular Approach for Clinical SLMs Driven by Synthetic Data with Pre-Instruction Tuning, Model Merging, and Clinical-Tasks Alignmentcited by 20
  21. 2024Read Like a Radiologist: Efficient Vision-Language Model for 3D Medical Imaging Interpretationcited by 18
  22. 2025Dream-VL & Dream-VLA: Open Vision-Language and Vision-Language-Action Models with Diffusion Language Model Backbonecited by 18
  23. 2025OmniV-Med: Scaling Medical Vision-Language Model for Universal Visual Understandingcited by 17
  24. 2024A Spectrum Evaluation Benchmark for Medical Multi-Modal Large Language Modelscited by 16
  25. 2025A Survey on Computational Pathology Foundation Models: Datasets, Adaptation Strategies, and Evaluation Taskscited by 16
  26. 2026MedMO: Grounding and Understanding Multimodal Large Language Model for Medical Imagescited by 15
  27. 2023Learning A Multi-Task Transformer Via Unified And Customized Instruction Tuning For Chest Radiograph Interpretationcited by 14
  28. 2025Towards Cardiac MRI Foundation Models: Comprehensive Visual-Tabular Representations for Whole-Heart Assessment and Beyondcited by 14
  29. 2024Medical Vision Generalist: Unifying Medical Imaging Tasks in Contextcited by 13
  30. 2023A Systematic Survey of Prompt Engineering on Vision-Language Foundation Modelscited by 12
  31. 2024Transcending Language Boundaries: Harnessing LLMs for Low-Resource Language Translationcited by 12
  32. 20253D Foundation Model for Generalizable Disease Detection in Head Computed Tomographycited by 12
  33. 2025Vision Language Models in Medicinecited by 12
  34. 2025UMIT: Unifying Medical Imaging Tasks via Vision-Language Modelscited by 12
  35. 2025A foundation model for human-AI collaboration in medical literature miningcited by 11
  36. 2025EndoMamba: An Efficient Foundation Model for Endoscopic Videos via Hierarchical Pre-trainingcited by 11
  37. 2024Conformal Generative Modeling with Improved Sample Efficiency through Sequential Greedy Filteringcited by 10
  38. 2025RetinalGPT: A Retinal Clinical Preference Conversational Assistant Powered by Large Vision-Language Modelscited by 10

50 most cited of 201 All 201 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2024MiniGPT-Med: Large Language Model as a General Interface for Radiology Diagnosis
  2. 2023XrayGPT: Chest Radiographs Summarization using Medical Vision-Language Models
  3. 2025A Large-Scale Vision-Language Dataset Derived from Open Scientific Literature to Advance Biomedical Generalist AI
  4. 2025Adapting Lightweight Vision Language Models for Radiological Visual Question Answering
  5. 2023Evaluation of ChatGPT Family of Models for Biomedical Reasoning and Classification
  6. 2024Multimodal Healthcare AI: Identifying and Designing Clinically Relevant Vision-Language Applications for Radiology
  7. 2023Exploring the Trade-Offs: Unified Large Language Models vs Local Fine-Tuned Models for Highly-Specific Radiology NLI Task
  8. 2023LLaVA-Med: Training a Large Language-and-Vision Assistant for Biomedicine in One Day