mlsanityData to 7 Oct 2026

HuatuoGPT-Vision, Towards Injecting Medical Visual Knowledge into Multimodal LLMs at Scale

arXiv 2406.19280 ↗First posted 27 Jun 2024cs.CV cs.AI cs.CLcited by 194

On the map: Multimodal LLMs / vision-language models → Medical vision-language models

Abstract (arXiv)

The rapid development of multimodal large language models (MLLMs), such as GPT-4V, has led to significant advancements. However, these models still face challenges in medical multimodal capabilities due to limitations in the quantity and quality of medical vision-text data, stemming from data privacy concerns and high annotation costs. While pioneering approaches utilize PubMed's large-scale, de-identified medical image-text pairs to address these limitations, they still fall short due to inherent data noise. To tackle this, we refined medical image-text pairs from PubMed and employed MLLMs (GPT-4V) in an 'unblinded' capacity to denoise and reformat the data, resulting in the creation of the PubMedVision dataset with 1.3 million medical VQA samples. Our validation demonstrates that: (1) PubMedVision can significantly enhance the medical multimodal capabilities of current MLLMs, showing significant improvement in benchmarks including the MMMU Health & Medicine track; (2) manual checks by medical experts and empirical results validate the superior data quality of our dataset compared to other data construction methods. Using PubMedVision, we train a 34B medical MLLM HuatuoGPT-Vision, which shows superior performance in medical multimodal scenarios among open-source MLLMs.

Key facts. Cited by 194 papers in the map (data through 7 Oct 2026). Placed in Medical vision-language models, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 26 of its 34 references that are in the map. First posted on arXiv 27 Jun 2024.

Builds on

26 of 34 references.

Closest ancestors · 4

Same direction, newest first

  1. 2024CARES: A Comprehensive Benchmark of Trustworthiness in Medical Vision Language Models10 Jun
  2. 2024OmniMedVQA: A New Large-Scale Comprehensive Evaluation Benchmark for Medical LVLM14 Feb
  3. 2023Multimodal ChatGPT for Medical Applications: an Experimental Study of GPT-4V29 Oct
  4. 2023LLaVA-Med: Training a Large Language-and-Vision Assistant for Biomedicine in One Day1 Jun

Other references · 22

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  2. 2019Sentence-BERT: Sentence Embeddings using Siamese BERT-Networkscited by 9,049
  3. 2023Visual Instruction Tuningcited by 7,792
  4. 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,069
  5. 2023Improved Baselines with Visual Instruction Tuningcited by 4,362
  6. 2022Flamingo: a Visual Language Model for Few-Shot Learningcited by 4,337
  7. 2023InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuningcited by 2,855
  8. 2023MMMU: A Massive Multi-discipline Multimodal Understanding and Reasoning Benchmark for Expert AGIcited by 2,044
  9. 2023mPLUG-Owl: Modularization Empowers Large Language Models with Multimodalitycited by 1,038
  10. 2023ShareGPT4V: Improving Large Multi-Modal Models with Better Captionscited by 1,000
  11. 2024Yi: Open Foundation Models by 01.AIcited by 706
  12. 2020PathVQA: 30000+ Questions for Medical Visual Question Answeringcited by 467
10 more
  1. 2021SLAKE: A Semantically-Labeled Knowledge-Enhanced Dataset for Medical Visual Question Answeringcited by 444
  2. 2023Med-Flamingo: a Multimodal Medical Few-shot Learnercited by 387
  3. 2023PMC-VQA: Visual Instruction Tuning for Medical Visual Question Answeringcited by 333
  4. 2023PMC-CLIP: Contrastive Language-Image Pre-training using Biomedical Documentscited by 212
  5. 2023HuatuoGPT, towards Taming Language Model to Be a Doctorcited by 211
  6. 2024ALLaVA: Harnessing GPT4V-Synthesized Data for Lite Vision-Language Modelscited by 204
  7. 2023Towards Generalist Foundation Model for Radiology by Leveraging Web-scale 2D&3D Medical Datacited by 176
  8. 2023HuatuoGPT-II, One-stage Training for Medical Adaption of LLMscited by 91
  9. 2024Apollo: A Lightweight Multilingual Medical LLM towards Democratizing Medical AI to 6B Peoplecited by 38
  10. 2024Reformatted Alignmentcited by 15

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 17; 2025: 106; 2026: 71.

194 papers.

Sort:

By area:

  1. 2024HuatuoGPT-o1, Towards Medical Complex Reasoning with LLMscited by 257
  2. 2025Lingshu: A Generalist Foundation Model for Unified Multimodal Medical Understanding and Reasoningcited by 217
  3. 2025MedVLM-R1: Incentivizing Medical Reasoning Capability of Vision-Language Models (VLMs) via Reinforcement Learningcited by 172
  4. 2025Med-R1: Reinforcement Learning for Generalizable Medical Reasoning in Vision-Language Modelscited by 144
  5. 2025HealthGPT: A Medical Large Vision-Language Model for Unifying Comprehension and Generation via Heterogeneous Knowledge Adaptationcited by 112
  6. 2024LongLLaVA: Scaling Multi-modal LLMs to 1000 Images Efficiently via a Hybrid Architecturecited by 100
  7. 2024MLLM can see? Dynamic Correction Decoding for Hallucination Mitigationcited by 87
  8. 2024MMed-RAG: Versatile Multimodal RAG System for Medical Vision Language Modelscited by 84
  9. 2025From AI for Science to Agentic Science: A Survey on Autonomous Scientific Discoverycited by 74
  10. 2024CoD, Towards an Interpretable Medical Agent using Chain of Diagnosiscited by 43
  11. 2025MAM: Modular Multi-Agent Framework for Multi-Modal Medical Diagnosis via Role-Specialized Collaborationcited by 39
  12. 2025MMedAgent-RL: Optimizing Multi-Agent Collaboration for Multimodal Medical Reasoningcited by 37
38 more
  1. 2025A Survey of Scientific Large Language Models: From Data Foundations to Agent Frontierscited by 34
  2. 2025Medical World Model: Generative Simulation of Tumor Evolution for Treatment Planningcited by 31
  3. 2024A Survey on Multimodal Benchmarks: In the Era of Large AI Modelscited by 27
  4. 2025Evidence-based diagnostic reasoning with multi-agent copilot for human pathologycited by 26
  5. 2025MedVLThinker: Simple Baselines for Multimodal Medical Reasoningcited by 26
  6. 2024Interpretable Bilingual Multimodal Large Language Model for Diverse Biomedical Taskscited by 24
  7. 2024GMAI-VL & GMAI-VL-5.5M: A Large Vision-Language Model and A Comprehensive Multimodal Dataset Towards General Medical AIcited by 23
  8. 2024ReflecTool: Towards Reflection-Aware Tool-Augmented Clinical Agentscited by 20
  9. 2024E3D-GPT: Enhanced 3D Visual Foundation for Medical Vision-Language Modelcited by 19
  10. 2025A Survey on Agentic Multimodal Large Language Modelscited by 19
  11. 2024Self-eXplainable AI for Medical Image Analysis: A Survey and New Outlookscited by 18
  12. 2025OmniV-Med: Scaling Medical Vision-Language Model for Universal Visual Understandingcited by 17
  13. 2025Citrus: Leveraging Expert Cognitive Pathways in a Medical Language Model for Advanced Medical Decision Supportcited by 16
  14. 2025Multimodal Large Language Models for Medicine: A Comprehensive Surveycited by 16
  15. 2025Improving Medical Reasoning with Curriculum-Aware Reinforcement Learningcited by 16
  16. 2025Chiron-o1: Igniting Multimodal Large Language Models towards Generalizable Medical Reasoning via Mentor-Intern Collaborative Searchcited by 16
  17. 2025Towards Better Dental AI: A Multimodal Benchmark and Instruction Dataset for Panoramic X-ray Analysiscited by 16
  18. 2024PA-LLaVA: A Large Language-Vision Assistant for Human Pathology Image Understandingcited by 15
  19. 2025MIMO: A medical vision language model with visual referring multimodal input and pixel grounding multimodal outputcited by 14
  20. 2025Med-CMR: A Fine-Grained Benchmark Integrating Visual Evidence and Clinical Logic for Medical Complex Multimodal Reasoningcited by 14
  21. 2024On Domain-Adaptive Post-Training for Multimodal Large Language Modelscited by 13
  22. 2025EndoBench: A Comprehensive Evaluation of Multi-Modal Large Language Models for Endoscopy Analysiscited by 13
  23. 2025Patho-AgenticRAG: Towards Multimodal Agentic Retrieval-Augmented Generation for Pathology VLMs via Reinforcement Learningcited by 13
  24. 2025DentVLM: A Multimodal Vision-Language Model for Comprehensive Dental Diagnosis and Enhanced Clinical Practicecited by 13
  25. 2025Think Twice to See More: Iterative Visual Reasoning in Medical VLMscited by 12
  26. 2025UniMedVL: Unifying Medical Multimodal Understanding and Generation through Observation-Knowledge-Analysiscited by 12
  27. 2026MedMO: Grounding and Understanding Multimodal Large Language Model for Medical Imagescited by 12
  28. 2025UMIT: Unifying Medical Imaging Tasks via Vision-Language Modelscited by 11
  29. 2024Beyond the Hype: A dispassionate look at vision-language models in medical scenariocited by 10
  30. 2025DrVD-Bench: Do Vision-Language Models Reason Like Human Doctors in Medical Image Diagnosis?cited by 10
  31. 2025MedReasoner: Reinforcement Learning Drives Reasoning Grounding from Clinical Thought to Pixel-Level Precisioncited by 10
  32. 2025OctoMed: Data Recipes for State-of-the-Art Multimodal Medical Reasoningcited by 10
  33. 2026IBISAgent: Reinforcing Pixel-Level Visual Reasoning in MLLMs for Universal Biomedical Object Referring and Segmentationcited by 10
  34. 2025MMXU: A Multi-Modal and Multi-X-ray Understanding Dataset for Disease Progressioncited by 9
  35. 2025RARL: Improving Medical VLM Reasoning and Generalization with Reinforcement Learning and LoRA under Data and Hardware Constraintscited by 9
  36. 2025Continual Learning for Generative AI: From LLMs to MLLMs and Beyondcited by 9
  37. 2026OmniCT: Towards a Unified Slice-Volume LVLM for Comprehensive CT Analysiscited by 9
  38. 2025MedSG-Bench: A Benchmark for Medical Image Sequences Groundingcited by 8

50 most cited of 194 All 194 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2024Parameter-Efficient Fine-Tuning Medical Multimodal Large Language Models for Medical Visual Grounding
  2. 2024PeFoMed: Parameter Efficient Fine-tuning of Multimodal Large Language Models for Medical Imaging
  3. 2026How Do Medical MLLMs Fail? A Study on Visual Grounding in Medical Images
  4. 2024From Text to Multimodality: Exploring the Evolution and Impact of Large Language Models in Medical Practice
  5. 2025Fleming-VL: Towards Universal Medical Visual Reasoning with Multimodal LLMs
  6. 2024An Early Investigation into the Utility of Multimodal Large Language Models in Medical Imaging
  7. 2023Qilin-Med-VL: Towards Chinese Large Vision-Language Model for General Healthcare
  8. 2024MultiMedEval: A Benchmark and a Toolkit for Evaluating Medical Vision-Language Models