# Medical Phrase Grounding with Region-Phrase Context Contrastive Alignment

arXiv: https://arxiv.org/abs/2303.07618 · HTML: https://mlsanity.com/paper/2303.07618/

## Key facts

- Cited by 32 papers in the map (data through 14 Oct 2026).
- Builds on 18 of its 24 references that are in the map.
- First posted on arXiv 14 Mar 2023.

## Builds on (most cited first, 18 of 18)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [PyTorch: An Imperative Style, High-Performance Deep Learning Library](https://mlsanity.com/paper/1912.01703/) (2019)
- [Representation Learning with Contrastive Predictive Coding](https://mlsanity.com/paper/1807.03748/) (2018)
- [End-to-End Object Detection with Transformers](https://mlsanity.com/paper/2005.12872/) (2020)
- [Fast R-CNN](https://mlsanity.com/paper/1504.08083/) (2015)
- [Generalized Intersection over Union: A Metric and A Loss for Bounding Box Regression](https://mlsanity.com/paper/1902.09630/) (2019)
- [ChestX-ray8: Hospital-scale Chest X-ray Database and Benchmarks on Weakly-Supervised Classification and Localization of Common Thorax Diseases](https://mlsanity.com/paper/1705.02315/) (2017)
- [MDETR -- Modulated Detection for End-to-End Multi-Modal Understanding](https://mlsanity.com/paper/2104.12763/) (2021)
- [MAttNet: Modular Attention Network for Referring Expression Comprehension](https://mlsanity.com/paper/1801.08186/) (2018)
- [Making the Most of Text Semantics to Improve Biomedical Vision--Language Processing](https://mlsanity.com/paper/2204.09817/) (2022)
- [TransVG: End-to-End Visual Grounding with Transformers](https://mlsanity.com/paper/2104.08541/) (2021)
- [A Fast and Accurate One-Stage Approach to Visual Grounding](https://mlsanity.com/paper/1908.06354/) (2019)
- [Referring Transformer: A One-step Approach to Multi-task Visual Grounding](https://mlsanity.com/paper/2106.03089/) (2021)
- [SeqTR: A Simple yet Universal Network for Visual Grounding](https://mlsanity.com/paper/2203.16265/) (2022)
- [Medical Image Understanding with Pretrained Vision Language Models: A Comprehensive Study](https://mlsanity.com/paper/2209.15517/) (2022)
- [Visual Grounding with Transformers](https://mlsanity.com/paper/2105.04281/) (2021)

## Built on by (most cited first, 30 of 32)

- [MAIRA-2: Grounded Radiology Report Generation](https://mlsanity.com/paper/2406.04449/) (2024)
- [Toward Visual Grounding: A Survey](https://mlsanity.com/paper/2412.20206/) (2024)
- [A Systematic Evaluation of GPT-4V's Multimodal Capability for Medical Image Analysis](https://mlsanity.com/paper/2310.20381/) (2023)
- [MedGround-R1: Advancing Medical Image Grounding via Spatial-Semantic Rewarded Group Relative Policy Optimization](https://mlsanity.com/paper/2507.02994/) (2025)
- [ChEX: Interactive Localization and Region Description in Chest X-rays](https://mlsanity.com/paper/2404.15770/) (2024)
- [M4CXR: Exploring Multi-task Potentials of Multi-modal Large Language Models for Chest X-ray Interpretation](https://mlsanity.com/paper/2408.16213/) (2024)
- [Uncertainty-aware Medical Diagnostic Phrase Identification and Grounding](https://mlsanity.com/paper/2404.06798/) (2024)
- [MedSG-Bench: A Benchmark for Medical Image Sequences Grounding](https://mlsanity.com/paper/2505.11852/) (2025)
- [Zero-Shot Medical Phrase Grounding with Off-the-shelf Diffusion Models](https://mlsanity.com/paper/2404.12920/) (2024)
- [A Chain of Diagnosis Framework for Accurate and Explainable Radiology Report Generation](https://mlsanity.com/paper/2508.09566/) (2025)
- [Med-GLIP: Advancing Medical Language-Image Pre-training with Large-scale Grounded Dataset](https://mlsanity.com/paper/2508.10528/) (2025)
- [A unified multi-task framework enables interpretable chest radiograph analysis](https://mlsanity.com/paper/2606.03417/) (2026)
- [Topicwise Separable Sentence Retrieval for Medical Report Generation](https://mlsanity.com/paper/2405.04175/) (2024)
- [Anatomical grounding pre-training for medical phrase grounding](https://mlsanity.com/paper/2502.16585/) (2025)
- [RadZero: Similarity-Based Cross-Attention for Explainable Vision-Language Alignment in Chest X-ray with Zero-Shot Multi-Task Capability](https://mlsanity.com/paper/2504.07416/) (2025)
- [Seeing the Trees for the Forest: Rethinking Weakly-Supervised Medical Visual Grounding](https://mlsanity.com/paper/2505.15123/) (2025)
- [PathVG: A New Benchmark and Dataset for Pathology Visual Grounding](https://mlsanity.com/paper/2502.20869/) (2025)
- [Generate to Ground: Multimodal Text Conditioning Boosts Phrase Grounding in Medical Vision-Language Models](https://mlsanity.com/paper/2507.12236/) (2025)
- [Phrase-grounded Fact-checking for Automatically Generated Chest X-ray Reports](https://mlsanity.com/paper/2509.21356/) (2025)
- [TRACE: Temporal Radiology with Anatomical Change Explanation for Grounded X-ray Report Generation](https://mlsanity.com/paper/2602.02963/) (2026)
- [Multi-label Instance-level Generalised Visual Grounding in Agriculture](https://mlsanity.com/paper/2603.06699/) (2026)
- [Anatomy-Grounded Weakly Supervised Prompt Tuning for Chest X-ray Latent Diffusion Models](https://mlsanity.com/paper/2506.10633/) (2025)
- [Multi Anatomy X-Ray Foundation Model](https://mlsanity.com/paper/2509.12146/) (2025)
- [Generalised Medical Phrase Grounding](https://mlsanity.com/paper/2512.01085/) (2025)
- [Enhancing Medical Visual Grounding via Knowledge-guided Spatial Prompts](https://mlsanity.com/paper/2604.01915/) (2026)
- [Concept-Guided Noisy Negative Suppression for Zero-Shot Classification and Grounding of Chest X-Ray Findings](https://mlsanity.com/paper/2605.19374/) (2026)
- [VICCA: Visual Interpretation and Comprehension of Chest X-ray Anomalies in Generated Report Without Human Feedback](https://mlsanity.com/paper/2501.17726/) (2025)
- [A Comparison of Object Detection and Phrase Grounding Models in Chest X-ray Abnormality Localization using Eye-tracking Data](https://mlsanity.com/paper/2503.01037/) (2025)
- [LoFi: Location-Aware Fine-Grained Representation Learning for Chest X-ray](https://mlsanity.com/paper/2603.19451/) (2026)
- [Rad-VLSM: A Cross-Modal Framework with Semantics-Assisted Prompting for Medical Segmentation and Diagnosis](https://mlsanity.com/paper/2605.18130/) (2026)

## Abstract (arXiv)

Medical phrase grounding (MPG) aims to locate the most relevant region in a medical image, given a phrase query describing certain medical findings, which is an important task for medical image analysis and radiological diagnosis. However, existing visual grounding methods rely on general visual features for identifying objects in natural images and are not capable of capturing the subtle and specialized features of medical findings, leading to sub-optimal performance in MPG. In this paper, we propose MedRPG, an end-to-end approach for MPG. MedRPG is built on a lightweight vision-language transformer encoder and directly predicts the box coordinates of mentioned medical findings, which can be trained with limited medical data, making it a valuable tool in medical image analysis. To enable MedRPG to locate nuanced medical findings with better region-phrase correspondences, we further propose Tri-attention Context contrastive alignment (TaCo). TaCo seeks context alignment to pull both the features and attention outputs of relevant region-phrase pairs close together while pushing those of irrelevant regions far away. This ensures that the final box prediction depends more on its finding-specific regions and phrases. Experimental results on three MPG datasets demonstrate that our MedRPG outperforms state-of-the-art visual grounding approaches by a large margin. Additionally, the proposed TaCo strategy is effective in enhancing finding localization ability and reducing spurious region-phrase correlations.