# Med-GLIP: Advancing Medical Language-Image Pre-training with Large-scale Grounded Dataset

arXiv: https://arxiv.org/abs/2508.10528 · HTML: https://mlsanity.com/paper/2508.10528/

## Key facts

- Cited by 7 papers in the map (data through 14 Oct 2026).
- Builds on 28 of its 41 references that are in the map.
- First posted on arXiv 14 Aug 2025.

## Builds on (most cited first, 28 of 28)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Segment Anything](https://mlsanity.com/paper/2304.02643/) (2023)
- [LLaVA-Med: Training a Large Language-and-Vision Assistant for Biomedicine in One Day](https://mlsanity.com/paper/2306.00890/) (2023)
- [Grounded Language-Image Pre-training](https://mlsanity.com/paper/2112.03857/) (2021)
- [PathVQA: 30000+ Questions for Medical Visual Question Answering](https://mlsanity.com/paper/2003.10286/) (2020)
- [SLAKE: A Semantically-Labeled Knowledge-Enhanced Dataset for Medical Visual Question Answering](https://mlsanity.com/paper/2102.09542/) (2021)
- [Generating Radiology Reports via Memory-driven Transformer](https://mlsanity.com/paper/2010.16056/) (2020)
- [DETRs with Collaborative Hybrid Assignments Training](https://mlsanity.com/paper/2211.12860/) (2022)
- [Segment Anything in Medical Images](https://mlsanity.com/paper/2304.12306/) (2023)
- [MedKLIP: Medical Knowledge Enhanced Language-Image Pre-Training in Radiology](https://mlsanity.com/paper/2301.02228/) (2023)
- [MedTrinity-25M: A Large-scale Multimodal Dataset with Multigranular Annotations for Medicine](https://mlsanity.com/paper/2408.02900/) (2024)
- [KiUT: Knowledge-injected U-Transformer for Radiology Report Generation](https://mlsanity.com/paper/2306.11345/) (2023)
- [TIGEr: Text-to-Image Grounding for Image Caption Evaluation](https://mlsanity.com/paper/1909.02050/) (2019)
- [A Comprehensive Survey of Foundation Models in Medicine](https://mlsanity.com/paper/2406.10729/) (2024)
- [Med-MoE: Mixture of Domain-Specific Experts for Lightweight Medical Vision-Language Models](https://mlsanity.com/paper/2404.10237/) (2024)
- [MedCoT: Medical Chain of Thought via Hierarchical Expert](https://mlsanity.com/paper/2412.13736/) (2024)
- [Enhanced Contrastive Learning with Multi-view Longitudinal Data for Chest X-ray Report Generation](https://mlsanity.com/paper/2502.20056/) (2025)
- [Medical Phrase Grounding with Region-Phrase Context Contrastive Alignment](https://mlsanity.com/paper/2303.07618/) (2023)
- [Interactive Medical Image Segmentation: A Benchmark Dataset and Baseline](https://mlsanity.com/paper/2411.12814/) (2024)
- [R-LLaVA: Improving Med-VQA Understanding through Visual Region of Interest](https://mlsanity.com/paper/2410.20327/) (2024)
- [Medical Multimodal Foundation Models in Clinical Diagnosis and Treatment: Applications, Challenges, and Future Directions](https://mlsanity.com/paper/2412.02621/) (2024)
- [Joint Visual and Text Prompting for Improved Object-Centric Perception with Multimodal Large Language Models](https://mlsanity.com/paper/2404.04514/) (2024)
- [OmniV-Med: Scaling Medical Vision-Language Model for Universal Visual Understanding](https://mlsanity.com/paper/2504.14692/) (2025)
- [V2T-CoT: From Vision to Text Chain-of-Thought for Medical Reasoning and Diagnosis](https://mlsanity.com/paper/2506.19610/) (2025)
- [Unsupervised Textual Grounding: Linking Words to Image Concepts](https://mlsanity.com/paper/1803.11185/) (2018)
- [KPL: Training-Free Medical Knowledge Mining of Vision-Language Models](https://mlsanity.com/paper/2501.11231/) (2025)
- [Med-U1: Incentivizing Unified Medical Reasoning in LLMs via Large-scale Reinforcement Learning](https://mlsanity.com/paper/2506.12307/) (2025)
- [Fair-MoE: Fairness-Oriented Mixture of Experts in Vision-Language Models](https://mlsanity.com/paper/2502.06094/) (2025)

## Built on by (most cited first, 7 of 7)

- [Ophiuchus: Incentivizing Tool-augmented "Think with Images" for Joint Medical Segmentation, Understanding and Reasoning](https://mlsanity.com/paper/2512.14157/) (2025)
- [MedGround: Bridging the Evidence Gap in Medical Vision-Language Models with Verified Grounding Data](https://mlsanity.com/paper/2601.06847/) (2026)
- [MedOpenClaw and MedFlowBench: Auditing Medical Agents in Full-Study Workflows](https://mlsanity.com/paper/2603.24649/) (2026)
- [Enhancing Medical Visual Grounding via Knowledge-guided Spatial Prompts](https://mlsanity.com/paper/2604.01915/) (2026)
- [Towards Responsible Multimodal Medical Reasoning via Context-Aligned Vision-Language Models](https://mlsanity.com/paper/2604.08815/) (2026)
- [BioMedVR: Confusion-Aware Mixture-of-Prompt Experts for Biomedical Visual Reprogramming](https://mlsanity.com/paper/2606.24740/) (2026)
- [MedPixel: A Unified Pixel-Language Model for Medical Reasoning and Segmentation](https://mlsanity.com/paper/2608.09818/) (2026)

## Abstract (arXiv)

Medical image grounding aims to align natural language phrases with specific regions in medical images, serving as a foundational task for intelligent diagnosis, visual question answering (VQA), and automated report generation (MRG). However, existing research is constrained by limited modality coverage, coarse-grained annotations, and the absence of a unified, generalizable grounding framework. To address these challenges, we construct a large-scale medical grounding dataset Med-GLIP-5M comprising over 5.3 million region-level annotations across seven imaging modalities, covering diverse anatomical structures and pathological findings. The dataset supports both segmentation and grounding tasks with hierarchical region labels, ranging from organ-level boundaries to fine-grained lesions. Based on this foundation, we propose Med-GLIP, a modality-aware grounding framework trained on Med-GLIP-5M. Rather than relying on explicitly designed expert modules, Med-GLIP implicitly acquires hierarchical semantic understanding from diverse training data -- enabling it to recognize multi-granularity structures, such as distinguishing lungs from pneumonia lesions. Extensive experiments demonstrate that Med-GLIP consistently outperforms state-of-the-art baselines across multiple grounding benchmarks. Furthermore, integrating its spatial outputs into downstream tasks, including medical VQA and report generation, leads to substantial performance gains. Our dataset will be released soon.