mlsanityData to 14 Oct 2026

Qilin-Med-VL: Towards Chinese Large Vision-Language Model for General Healthcare

arXiv 2310.17956 ↗First posted 27 Oct 2023cs.CV cs.AI cs.CLcited by 62

On the map: Multimodal LLMs / vision-language models → Medical vision-language models

Abstract (arXiv)

Large Language Models (LLMs) have introduced a new era of proficiency in comprehending complex healthcare and biomedical topics. However, there is a noticeable lack of models in languages other than English and models that can interpret multi-modal input, which is crucial for global healthcare accessibility. In response, this study introduces Qilin-Med-VL, the first Chinese large vision-language model designed to integrate the analysis of textual and visual data. Qilin-Med-VL combines a pre-trained Vision Transformer (ViT) with a foundational LLM. It undergoes a thorough two-stage curriculum training process that includes feature alignment and instruction tuning. This method enhances the model's ability to generate medical captions and answer complex medical queries. We also release ChiMed-VL, a dataset consisting of more than 1M image-text pairs. This dataset has been carefully curated to enable detailed and comprehensive interpretation of medical data using various types of images.

Key facts. Cited by 62 papers in the map (data through 14 Oct 2026). Placed in Medical vision-language models, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 14 of its 31 references that are in the map. First posted on arXiv 27 Oct 2023.

Builds on

14 of 31 references.

Closest ancestors · 2

Same direction, newest first

  1. 2023LLaVA-Med: Training a Large Language-and-Vision Assistant for Biomedicine in One Day1 Jun
  2. 2023MedBLIP: Bootstrapping Language-Image Pre-training from 3D Medical Images and Texts18 May

Other references · 12

  1. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 30,186
  2. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,863
  3. 2023Improved Baselines with Visual Instruction Tuningcited by 4,418
  4. 2023Capabilities of GPT-4 on Medical Challenge Problemscited by 638
  5. 2023Med-Flamingo: a Multimodal Medical Few-shot Learnercited by 392
  6. 2023InternLM-XComposer: A Vision-Language Large Model for Advanced Text-image Comprehension and Compositioncited by 297
  7. 2023PMC-CLIP: Contrastive Language-Image Pre-training using Biomedical Documentscited by 212
  8. 2023Towards Generalist Foundation Model for Radiology by Leveraging Web-scale 2D&3D Medical Datacited by 179
  9. 2023Large Multilingual Models Pivot Zero-Shot Multimodal Learning across Languagescited by 52
  10. 2023Radiology-Llama2: Best-in-Class Large Language Model for Radiologycited by 42
  11. 2022GreenPLM: Cross-Lingual Transfer of Monolingual Pre-Trained Language Models at Almost No Costcited by 24
  12. 2023Qilin-Med: Multi-stage Knowledge Injection Advanced Medical Large Language Modelcited by 24

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 6; 2024: 30; 2025: 20; 2026: 6.

62 papers.

Sort:

By area:

  1. 2024The Revolution of Multimodal Large Language Models: A Surveycited by 137
  2. 2023A Survey of Large Language Models in Medicine: Progress, Application, and Challengecited by 124
  3. 2024A Comprehensive Survey of Scientific Large Language Models and Their Applications in Scientific Discoverycited by 96
  4. 2024GMAI-MMBench: A Comprehensive Multimodal Evaluation Benchmark Towards General Medical AIcited by 96
  5. 2024Mini-InternVL: A Flexible-Transfer Pocket Multimodal Model with 5% Parameters and 90% Performancecited by 94
  6. 2024Foundation Model for Advancing Healthcare: Challenges, Opportunities, and Future Directionscited by 77
  7. 2024A Comprehensive Survey of Foundation Models in Medicinecited by 56
  8. 2024A Survey of Mathematical Reasoning in the Era of Multimodal Large Language Model: Benchmark, Method & Challengescited by 55
  9. 2024FinTextQA: A Dataset for Long-form Financial Question Answeringcited by 54
  10. 2024A Comprehensive Survey of Large Language Models and Multimodal Large Language Models in Medicinecited by 45
  11. 2025Improving Medical Large Vision-Language Models with Abnormal-Aware Feedbackcited by 42
  12. 2023CLIP in Medical Imaging: A Surveycited by 40
38 more
  1. 2025MMedAgent-RL: Optimizing Multi-Agent Collaboration for Multimodal Medical Reasoningcited by 40
  2. 2025MAM: Modular Multi-Agent Framework for Multi-Modal Medical Diagnosis via Role-Specialized Collaborationcited by 40
  3. 2025A Survey of Scientific Large Language Models: From Data Foundations to Agent Frontierscited by 37
  4. 2024A Survey on Medical Large Language Models: Technology, Application, Trustworthiness, and Future Directionscited by 34
  5. 2023Visual Instruction Tuning towards General-Purpose Multimodal Model: A Surveycited by 29
  6. 2024Data-Centric Foundation Models in Computational Healthcare: A Surveycited by 27
  7. 2024Interpretable Bilingual Multimodal Large Language Model for Diverse Biomedical Taskscited by 24
  8. 2024GMAI-VL & GMAI-VL-5.5M: A Large Vision-Language Model and A Comprehensive Multimodal Dataset Towards General Medical AIcited by 24
  9. 2024Look, Compare, Decide: Alleviating Hallucination in Large Vision-Language Models via Multi-View Multi-Path Reasoningcited by 23
  10. 2024PhysGame: Uncovering Physical Commonsense Violations in Gameplay Videoscited by 22
  11. 2024Has Multimodal Learning Delivered Universal Intelligence in Healthcare? A Comprehensive Surveycited by 20
  12. 2024Fine-Grained Verifiers: Preference Modeling as Next-token Prediction in Vision-Language Alignmentcited by 19
  13. 2024Medical Multimodal Foundation Models in Clinical Diagnosis and Treatment: Applications, Challenges, and Future Directionscited by 18
  14. 2024Large Language Model Benchmarks in Medical Taskscited by 17
  15. 2024RAP: Efficient Text-Video Retrieval with Sparse-and-Correlated Adaptercited by 16
  16. 2025Multimodal Large Language Models for Medicine: A Comprehensive Surveycited by 16
  17. 2025QSVD: Efficient Low-rank Approximation for Unified Query-Key-Value Weight Compression in Low-Precision Vision-Language Modelscited by 16
  18. 2024Mind the Modality Gap: Towards a Remote Sensing Vision-Language Model via Cross-modal Alignmentcited by 15
  19. 2026MedMO: Grounding and Understanding Multimodal Large Language Model for Medical Imagescited by 15
  20. 2023Adapting Large Language Models for Education: Foundational Capabilities, Potentials, and Challengescited by 13
  21. 2023Exploring Recommendation Capabilities of GPT-4V(ision): A Preliminary Case Studycited by 12
  22. 2025UMIT: Unifying Medical Imaging Tasks via Vision-Language Modelscited by 12
  23. 2024Dr-LLaVA: Visual Instruction Tuning with Symbolic Clinical Groundingcited by 10
  24. 2024Medical MLLM is Vulnerable: Cross-Modality Jailbreak and Mismatched Attacks on Medical Multimodal Large Language Modelscited by 10
  25. 2025EchoBench: Benchmarking Sycophancy in Medical Large Vision-Language Modelscited by 9
  26. 2024RITUAL: Random Image Transformations as a Universal Anti-hallucination Lever in Large Vision Language Modelscited by 8
  27. 2025Foundation Models in Medical Imaging: A Review and Outlookcited by 8
  28. 2024MUSE: Mamba is Efficient Multi-scale Learner for Text-video Retrievalcited by 7
  29. 2025GAIA: A Global, Multi-modal, Multi-scale Vision-Language Dataset for Remote Sensing Image Analysiscited by 7
  30. 2025PET2Rep: Towards Vision-Language Model-Drived Automated Radiology Report Generation for Positron Emission Tomographycited by 7
  31. 2025Toward a Vision-Language Foundation Model for Medical Data: Multimodal Dataset and Benchmarks for Vietnamese PET/CT Report Generationcited by 7
  32. 2025Elicit and Enhance: Advancing Multimodal Reasoning in Medical Scenarioscited by 6
  33. 2025FunBench: Benchmarking Fundus Reading Skills of MLLMscited by 5
  34. 2026MedGround: Bridging the Evidence Gap in Medical Vision-Language Models with Verified Grounding Datacited by 5
  35. 2024Semi-Supervised Multimodal Multi-Instance Learning for Aortic Stenosis Diagnosiscited by 4
  36. 2025LVLM-Compress-Bench: Benchmarking the Broader Impact of Large Vision-Language Model Compressioncited by 4
  37. 2023How to Bridge the Gap between Modalities: Survey on Multimodal Large Language Modelcited by 3
  38. 2024Mitigating Multilingual Hallucination in Large Vision-Language Modelscited by 3

50 most cited of 62 All 62 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2024LLaVA-Ultra: Large Chinese Language and Vision Assistant for Ultrasound
  2. 2024On Large Visual Language Models for Medical Imaging Analysis: An Empirical Study
  3. 2025Fleming-VL: Towards Universal Medical Visual Reasoning with Multimodal LLMs
  4. 2025MedM-VL: What Makes a Good Medical LVLM?
  5. 2025Vision Language Models in Medicine
  6. 2025Medical Large Vision Language Models with Multi-Image Visual Ability
  7. 2026A visual large language foundational model for medical image recognition using clinician-contributed online resources
  8. 2024HuatuoGPT-Vision, Towards Injecting Medical Visual Knowledge into Multimodal LLMs at Scale