mlsanityData to 7 Oct 2026

A Refer-and-Ground Multimodal Large Language Model for Biomedicine

arXiv 2406.18146 ↗First posted 26 Jun 2024cs.CVcited by 14

On the map: Multimodal LLMs / vision-language models → Medical vision-language models

Abstract (arXiv)

With the rapid development of multimodal large language models (MLLMs), especially their capabilities in visual chat through refer and ground functionalities, their significance is increasingly recognized. However, the biomedical field currently exhibits a substantial gap in this area, primarily due to the absence of a dedicated refer and ground dataset for biomedical images. To address this challenge, we devised the Med-GRIT-270k dataset. It comprises 270k question-and-answer pairs and spans eight distinct medical imaging modalities. Most importantly, it is the first dedicated to the biomedical domain and integrating refer and ground conversations. The key idea is to sample large-scale biomedical image-mask pairs from medical segmentation datasets and generate instruction datasets from text using chatGPT. Additionally, we introduce a Refer-and-Ground Multimodal Large Language Model for Biomedicine (BiRD) by using this dataset and multi-task instruction learning. Extensive experiments have corroborated the efficacy of the Med-GRIT-270k dataset and the multi-modal, fine-grained interactive capabilities of the BiRD model. This holds significant reference value for the exploration and development of intelligent biomedical assistants.

Key facts. Cited by 14 papers in the map (data through 7 Oct 2026). Placed in Medical vision-language models, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 19 of its 29 references that are in the map. First posted on arXiv 26 Jun 2024.

Builds on

19 of 29 references.

Closest ancestors · 1

Same direction, newest first

  1. 2023LLaVA-Med: Training a Large Language-and-Vision Assistant for Biomedicine in One Day1 Jun

Other references · 18

  1. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  2. 2023Visual Instruction Tuningcited by 7,792
  3. 2023Qwen Technical Reportcited by 3,273
  4. 2023Qwen-VL: A Versatile Vision-Language Model for Understanding, Localization, Text Reading, and Beyondcited by 2,179
  5. 2016SPICE: Semantic Propositional Image Caption Evaluationcited by 820
  6. 2023MiniGPT-v2: large language model as a unified interface for vision-language multi-task learningcited by 617
  7. 2023NExT-GPT: Any-to-Any Multimodal LLMcited by 610
  8. 2022MedCLIP: Contrastive Learning from Unpaired Medical Images and Textcited by 564
  9. 2023Ferret: Refer and Ground Anything Anywhere at Any Granularitycited by 498
  10. 2023GPT4RoI: Instruction Tuning Large Language Model on Region-of-Interestcited by 300
  11. 2023Towards Generalist Biomedical AIcited by 286
  12. 2024AnyGPT: Unified Multimodal LLM with Discrete Sequence Modelingcited by 213
6 more
  1. 2023Towards Generalist Foundation Model for Radiology by Leveraging Web-scale 2D&3D Medical Datacited by 176
  2. 2022LViT: Language meets Vision Transformer in Medical Image Segmentationcited by 110
  3. 2023BioMedGPT: Open Multimodal Generative Pre-trained Transformer for BioMedicinecited by 99
  4. 2023SA-Med2D-20M Dataset: Segment Anything in 2D Medical Imaging with 20 Million maskscited by 54
  5. 2024Cross-Modal Conditioned Reconstruction for Language-guided Medical Image Segmentationcited by 27
  6. 2024SegICL: A Multimodal In-context Learning Framework for Enhanced Segmentation in Medical Imagingcited by 5

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 2; 2025: 9; 2026: 3.

14 papers.

Sort:

By area:

  1. 2024Large Language Model with Region-guided Referring and Grounding for CT Report Generationcited by 27
  2. 2024Interpretable Bilingual Multimodal Large Language Model for Diverse Biomedical Taskscited by 24
  3. 2025MedGround-R1: Advancing Medical Image Grounding via Spatial-Semantic Rewarded Group Relative Policy Optimizationcited by 23
  4. 2025AOR: Anatomical Ontology-Guided Reasoning for Medical Large Multimodal Model in Chest X-Ray Interpretationcited by 8
  5. 2025Region-Aware Multimodal Large Language Model via SlowFast Tokenization and Pseudo-Mask Guidance for 3D CT Report Generationcited by 6
  6. 2026MedGround: Bridging the Evidence Gap in Medical Vision-Language Models with Verified Grounding Datacited by 5
  7. 2025RAU: Reference-based Anatomical Understanding with Vision Language Modelscited by 3
  8. 2025MedReason-R1: Learning to Reason for CT Diagnosis with Reinforcement Learning and Local Zoomcited by 3
  9. 2025ClinKD: Cross-Modal Clinical Knowledge Distiller For Multi-Task Medical Imagescited by 2
  10. 2025PathVG: A New Benchmark and Dataset for Pathology Visual Groundingcited by 2
  11. 2025Multi-Modal Explainable Medical AI Assistant for Trustworthy Human-AI Collaborationcited by 2
  12. 2026Step-CoT: Stepwise Visual Chain-of-Thought for Medical Visual Question Answeringcited by 2
2 more
  1. 2026Aloe-Vision: Robust Vision-Language Models for Healthcarecited by 1
  2. 2025EVLF-FM: Explainable Vision Language Foundation Model for Medicinecited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024Towards a Multimodal Large Language Model with Pixel-Level Insight for Biomedicine
  2. 2024PeFoMed: Parameter Efficient Fine-tuning of Multimodal Large Language Models for Medical Imaging
  3. 2025MIMO: A medical vision language model with visual referring multimodal input and pixel grounding multimodal output
  4. 2024Advancing High Resolution Vision-Language Models in Biomedicine
  5. 2024Parameter-Efficient Fine-Tuning Medical Multimodal Large Language Models for Medical Visual Grounding
  6. 2025BIOMEDICA: An Open Biomedical Image-Caption Archive, Dataset, and Vision-Language Models Derived from Scientific Literature
  7. 2025LLaDA-MedV: Exploring Large Language Diffusion Models for Biomedical Image Understanding
  8. 2025Fleming-VL: Towards Universal Medical Visual Reasoning with Multimodal LLMs