# Align2Ground: Weakly Supervised Phrase Grounding Guided by Image-Caption Alignment

arXiv: https://arxiv.org/abs/1903.11649 · HTML: https://mlsanity.com/paper/1903.11649/

## Key facts

- Cited by 51 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 27 Mar 2019.

## Built on by (most cited first, 30 of 51)

- [Making the Most of Text Semantics to Improve Biomedical Vision--Language Processing](https://mlsanity.com/paper/2204.09817/) (2022)
- [Toward Visual Grounding: A Survey](https://mlsanity.com/paper/2412.20206/) (2024)
- [Relation-aware Instance Refinement for Weakly Supervised Visual Grounding](https://mlsanity.com/paper/2103.12989/) (2021)
- [CLIP-VG: Self-paced Curriculum Adapting of CLIP for Visual Grounding](https://mlsanity.com/paper/2305.08685/) (2023)
- [Referring Image Segmentation Using Text Supervision](https://mlsanity.com/paper/2308.14575/) (2023)
- [Refer-it-in-RGBD: A Bottom-up Approach for 3D Visual Grounding in RGBD Images](https://mlsanity.com/paper/2103.07894/) (2021)
- [Pseudo-Q: Generating Pseudo Language Queries for Visual Grounding](https://mlsanity.com/paper/2203.08481/) (2022)
- [Vision-Language Intelligence: Tasks, Representation Learning, and Large Models](https://mlsanity.com/paper/2203.01922/) (2022)
- [Detailed Annotations of Chest X-Rays via CT Projection for Report Understanding](https://mlsanity.com/paper/2210.03416/) (2022)
- [Shatter and Gather: Learning Referring Image Segmentation with Text Supervision](https://mlsanity.com/paper/2308.15512/) (2023)
- [Separating Skills and Concepts for Novel Visual Question Answering](https://mlsanity.com/paper/2107.09106/) (2021)
- [Improving Visual Grounding by Encouraging Consistent Gradient-based Explanations](https://mlsanity.com/paper/2206.15462/) (2022)
- [Improving Pneumonia Localization via Cross-Attention on Medical Images and Reports](https://mlsanity.com/paper/2110.03094/) (2021)
- [2D Human Pose Estimation: A Survey](https://mlsanity.com/paper/2204.07370/) (2022)
- [Referring to Any Person](https://mlsanity.com/paper/2503.08507/) (2025)
- [Look at What I'm Doing: Self-Supervised Spatial Grounding of Narrations in Instructional Videos](https://mlsanity.com/paper/2110.10596/) (2021)
- [Contextual Modeling for 3D Dense Captioning on Point Clouds](https://mlsanity.com/paper/2210.03925/) (2022)
- [Distilling Coarse-to-Fine Semantic Matching Knowledge for Weakly Supervised 3D Visual Grounding](https://mlsanity.com/paper/2307.09267/) (2023)
- [Improved Visual Grounding through Self-Consistent Explanations](https://mlsanity.com/paper/2312.04554/) (2023)
- [Detector-Free Weakly Supervised Grounding by Separation](https://mlsanity.com/paper/2104.09829/) (2021)
- [Who's Waldo? Linking People Across Text and Images](https://mlsanity.com/paper/2108.07253/) (2021)
- [Weakly-Supervised Video Object Grounding via Causal Intervention](https://mlsanity.com/paper/2112.00475/) (2021)
- [AircraftVerse: A Large-Scale Multimodal Dataset of Aerial Vehicle Designs](https://mlsanity.com/paper/2306.05562/) (2023)
- [CTRL-O: Language-Controllable Object-Centric Visual Representation Learning](https://mlsanity.com/paper/2503.21747/) (2025)
- [STPro: Spatial and Temporal Progressive Learning for Weakly Supervised Spatio-Temporal Grounding](https://mlsanity.com/paper/2502.20678/) (2025)
- [Contextual Self-paced Learning for Weakly Supervised Spatio-Temporal Video Grounding](https://mlsanity.com/paper/2501.17053/) (2025)
- [Weakly Supervised Grounding for VQA in Vision-Language Transformers](https://mlsanity.com/paper/2207.02334/) (2022)
- [Who are you referring to? Coreference resolution in image narrations](https://mlsanity.com/paper/2211.14563/) (2022)
- [OV-VG: A Benchmark for Open-Vocabulary Visual Grounding](https://mlsanity.com/paper/2310.14374/) (2023)
- [Learning from Synthetic Data for Visual Grounding](https://mlsanity.com/paper/2403.13804/) (2024)

## Abstract (arXiv)

We address the problem of grounding free-form textual phrases by using weak supervision from image-caption pairs. We propose a novel end-to-end model that uses caption-to-image retrieval as a `downstream' task to guide the process of phrase localization. Our method, as a first step, infers the latent correspondences between regions-of-interest (RoIs) and phrases in the caption and creates a discriminative image representation using these matched RoIs. In a subsequent step, this (learned) representation is aligned with the caption. Our key contribution lies in building this `caption-conditioned' image encoding which tightly couples both the tasks and allows the weak supervision to effectively guide visual grounding. We provide an extensive empirical and qualitative analysis to investigate the different components of our proposed model and compare it with competitive baselines. For phrase localization, we report an improvement of 4.9% (absolute) over the prior state-of-the-art on the VisualGenome dataset. We also report results that are at par with the state-of-the-art on the downstream caption-to-image retrieval task on COCO and Flickr30k datasets.