mlsanityData to 7 Oct 2026

LLaVA-Med: Training a Large Language-and-Vision Assistant for Biomedicine in One Day

arXiv 2306.00890 ↗First posted 1 Jun 2023cs.CV cs.CLcited by 1,287

On the map: Multimodal LLMs / vision-language models → Medical vision-language models

Abstract (arXiv)

Conversational generative AI has demonstrated remarkable promise for empowering biomedical practitioners, but current investigations focus on unimodal text. Multimodal conversational AI has seen rapid progress by leveraging billions of image-text pairs from the public web, but such general-domain vision-language models still lack sophistication in understanding and conversing about biomedical images. In this paper, we propose a cost-efficient approach for training a vision-language conversational assistant that can answer open-ended research questions of biomedical images. The key idea is to leverage a large-scale, broad-coverage biomedical figure-caption dataset extracted from PubMed Central, use GPT-4 to self-instruct open-ended instruction-following data from the captions, and then fine-tune a large general-domain vision-language model using a novel curriculum learning method. Specifically, the model first learns to align biomedical vocabulary using the figure-caption pairs as is, then learns to master open-ended conversational semantics using GPT-4 generated instruction-following data, broadly mimicking how a layperson gradually acquires biomedical knowledge. This enables us to train a Large Language and Vision Assistant for BioMedicine (LLaVA-Med) in less than 15 hours (with eight A100s). LLaVA-Med exhibits excellent multimodal conversational capability and can follow open-ended instruction to assist with inquiries about a biomedical image. On three standard biomedical visual question answering datasets, LLaVA-Med outperforms previous supervised state-of-the-art on certain metrics. To facilitate biomedical multimodal research, we will release our instruction-following data and the LLaVA-Med model.

Key facts. Cited by 1,287 papers in the map (data through 7 Oct 2026). Placed in Medical vision-language models, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 27 of its 56 references that are in the map. First posted on arXiv 1 Jun 2023.

Builds on

27 of 56 references.

Closest ancestors · 1

Same direction, newest first

  1. 2022Self-supervised vision-language pretraining for Medical visual question answering24 Nov

Other references · 26

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  2. 2023GPT-4 Technical Reportcited by 18,132
  3. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  4. 2020Retrieval-Augmented Generation for Knowledge-Intensive NLP Taskscited by 9,160
  5. 2023Visual Instruction Tuningcited by 7,792
  6. 2023G-Eval: NLG Evaluation using GPT-4 with Better Human Alignmentcited by 2,205
  7. 2019BioBERT: a pre-trained biomedical language representation model for biomedical text miningcited by 1,935
  8. 2022Learn to Explain: Multimodal Reasoning via Thought Chains for Science Question Answeringcited by 1,881
  9. 2020Don't Stop Pretraining: Adapt Language Models to Domains and Taskscited by 1,548
  10. 2021A General Language Assistant as a Laboratory for Alignmentcited by 1,023
  11. 2020Domain-Specific Language Model Pretraining for Biomedical Natural Language Processingcited by 996
  12. 2023Instruction Tuning with GPT-4cited by 676
14 more
  1. 2023Capabilities of GPT-4 on Medical Challenge Problemscited by 630
  2. 2022BioGPT: Generative Pre-trained Transformer for Biomedical Text Generation and Miningcited by 531
  3. 2019ClinicalBERT: Modeling Clinical Notes and Predicting Hospital Readmissioncited by 487
  4. 2020PathVQA: 30000+ Questions for Medical Visual Question Answeringcited by 467
  5. 2021SLAKE: A Semantically-Labeled Knowledge-Enhanced Dataset for Medical Visual Question Answeringcited by 444
  6. 2022Making the Most of Text Semantics to Improve Biomedical Vision--Language Processingcited by 336
  7. 2023MedAlpaca -- An Open-Source Collection of Medical Conversational AI Models and Training Datacited by 282
  8. 2023HuaTuo: Tuning LLaMA Model with Chinese Medical Knowledgecited by 161
  9. 2022ELEVATER: A Benchmark and Toolkit for Evaluating Language-Augmented Visual Modelscited by 141
  10. 2023DoctorGLM: Fine-tuning your Chinese Doctor is not a Herculean Taskcited by 130
  11. 2022Vision-Language Pre-training: Basics, Recent Advances, and Future Trendscited by 117
  12. 2023Learning Customized Visual Models with Retrieval-Augmented Knowledgecited by 58
  13. 2023Open-Ended Medical Visual Question Answering Through Prefix Tuning of Language Modelscited by 58
  14. 2023Q2ATransformer: Improving Medical VQA via an Answer Querying Decodercited by 31

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 70; 2024: 320; 2025: 474; 2026: 423.

1,287 papers.

Sort:

By area:

  1. 2023Improved Baselines with Visual Instruction Tuningcited by 4,362
  2. 2023A Survey on Multimodal Large Language Modelscited by 898
  3. 2024VideoLLaMA 2: Advancing Spatial-Temporal Modeling and Audio Understanding in Video-LLMscited by 749
  4. 2025VideoLLaMA 3: Frontier Multimodal Foundation Models for Image and Video Understandingcited by 502
  5. 2023OPERA: Alleviating Hallucination in Multi-Modal Large Language Models via Over-Trust Penalty and Retrospection-Allocationcited by 468
  6. 2023DriveGPT4: Interpretable End-to-end Autonomous Driving via Large Language Modelcited by 455
  7. 2025DeepEyes: Incentivizing "Thinking with Images" via Reinforcement Learningcited by 398
  8. 2023Med-Flamingo: a Multimodal Medical Few-shot Learnercited by 387
  9. 2025HealthBench: Evaluating Large Language Models Towards Improved Human Healthcited by 385
  10. 2023Monkey: Image Resolution and Text Label Are Important Things for Large Multi-modal Modelscited by 356
  11. 2023GeoChat: Grounded Large Vision-Language Model for Remote Sensingcited by 320
  12. 2023MedAgents: Large Language Models as Collaborators for Zero-shot Medical Reasoningcited by 311
38 more
  1. 2025R1-VL: Learning to Reason with Multimodal Large Language Models via Step-wise Group Relative Policy Optimizationcited by 296
  2. 2023Towards Generalist Biomedical AIcited by 286
  3. 2024MM-LLMs: Recent Advances in MultiModal Large Language Modelscited by 277
  4. 2023Multimodal Foundation Models: From Specialists to General-Purpose Assistantscited by 247
  5. 2023LVLM-eHub: A Comprehensive Evaluation Benchmark for Large Vision-Language Modelscited by 217
  6. 2025Lingshu: A Generalist Foundation Model for Unified Multimodal Medical Understanding and Reasoningcited by 217
  7. 2023MetaTool Benchmark for Large Language Models: Deciding Whether to Use Tools and Which to Usecited by 209
  8. 2023Video Understanding with Large Language Models: A Surveycited by 203
  9. 2024Aligning Modalities in Vision Large Language Models via Preference Fine-tuningcited by 197
  10. 2024HuatuoGPT-Vision, Towards Injecting Medical Visual Knowledge into Multimodal LLMs at Scalecited by 194
  11. 2024Generalist Foundation Models from a Multimodal Dataset for 3D Computed Tomographycited by 192
  12. 2024OmniMedVQA: A New Large-Scale Comprehensive Evaluation Benchmark for Medical LVLMcited by 182
  13. 2024EarthGPT: A Universal Multi-modal Large Language Model for Multi-sensor Image Comprehension in Remote Sensing Domaincited by 181
  14. 2023Towards Generalist Foundation Model for Radiology by Leveraging Web-scale 2D&3D Medical Datacited by 176
  15. 2025Multimodal Chain-of-Thought Reasoning: A Comprehensive Surveycited by 176
  16. 2025MLLMs Know Where to Look: Training-free Perception of Small Visual Details with Multimodal LLMscited by 172
  17. 2025MedVLM-R1: Incentivizing Medical Reasoning Capability of Vision-Language Models (VLMs) via Reinforcement Learningcited by 172
  18. 2023MMICL: Empowering Vision-language Model with Multi-Modal In-Context Learningcited by 171
  19. 2024Agent AI: Surveying the Horizons of Multimodal Interactioncited by 167
  20. 2024Ovis: Structural Embedding Alignment for Multimodal Large Language Modelcited by 167
  21. 2023A Survey of Large Language Models for Healthcare: from Data, Technology, and Applications to Accountability and Ethicscited by 162
  22. 2024M3D: Advancing 3D Medical Image Analysis with Multi-Modal Large Language Modelscited by 154
  23. 2025Med-R1: Reinforcement Learning for Generalizable Medical Reasoning in Vision-Language Modelscited by 144
  24. 2023To See is to Believe: Prompting GPT-4V for Better Visual Instruction Tuningcited by 143
  25. 2024Multimodal ArXiv: A Dataset for Improving Scientific Comprehension of Large Vision-Language Modelscited by 138
  26. 2025Magma: A Foundation Model for Multimodal AI Agentscited by 138
  27. 2023On the Challenges and Perspectives of Foundation Models for Medical Image Analysiscited by 135
  28. 2024The Revolution of Multimodal Large Language Models: A Surveycited by 134
  29. 2024LHRS-Bot: Empowering Remote Sensing with VGI-Enhanced Large Multimodal Language Modelcited by 133
  30. 2024LLaVA-Critic: Learning to Evaluate Multimodal Modelscited by 132
  31. 2024Exploring scalable medical image encoders beyond text supervisioncited by 129
  32. 2023Silkie: Preference Distillation for Large Visual Language Modelscited by 128
  33. 2023Dolphins: Multimodal Language Model for Drivingcited by 124
  34. 2023A Survey of Large Language Models in Medicine: Progress, Application, and Challengecited by 123
  35. 2024Vision-Language Models for Medical Report Generation and Visual Question Answering: A Reviewcited by 121
  36. 2024CVQA: Culturally-diverse Multilingual Visual Question Answering Benchmarkcited by 118
  37. 2023Towards Accurate Differential Diagnosis with Large Language Modelscited by 115
  38. 2023LLaVA-Grounding: Grounded Visual Chat with Large Multimodal Modelscited by 113

50 most cited of 1,287 All 1,287 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2025No Tokens Wasted: Leveraging Long Context in Biomedical Vision-Language Models
  2. 2024A Survey for Large Language Models in Biomedicine
  3. 2026OpenMedReason: Scientific Reasoning Supervision for Medical Vision-Language Models
  4. 2023BiomedGPT: A Generalist Vision-Language Foundation Model for Diverse Biomedical Tasks
  5. 2024LLaVA-Ultra: Large Chinese Language and Vision Assistant for Ultrasound
  6. 2025Advancing AI Research Assistants with Expert-Involved Learning
  7. 2025BIOMEDICA: An Open Biomedical Image-Caption Archive, Dataset, and Vision-Language Models Derived from Scientific Literature
  8. 2024LLaVA-Surg: Towards Multimodal Surgical Assistant via Structured Surgical Video Learning