# Phrase Localization and Visual Relationship Detection with Comprehensive Image-Language Cues

arXiv: https://arxiv.org/abs/1611.06641 · HTML: https://mlsanity.com/paper/1611.06641/

## Key facts

- Cited by 33 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 21 Nov 2016.

## Built on by (most cited first, 30 of 33)

- [MDETR -- Modulated Detection for End-to-End Multi-Modal Understanding](https://mlsanity.com/paper/2104.12763/) (2021)
- [Multi3DRefer: Grounding Text Description to Multiple 3D Objects](https://mlsanity.com/paper/2309.05251/) (2023)
- [A Comprehensive Survey of Scene Graphs: Generation and Application](https://mlsanity.com/paper/2104.01111/) (2021)
- [Scene Graph Generation: A Comprehensive Survey](https://mlsanity.com/paper/2201.00443/) (2022)
- [Fully Convolutional Scene Graph Generation](https://mlsanity.com/paper/2103.16083/) (2021)
- [Toward Visual Grounding: A Survey](https://mlsanity.com/paper/2412.20206/) (2024)
- [Towards General Purpose Vision Systems](https://mlsanity.com/paper/2104.00743/) (2021)
- [CoTDet: Affordance Knowledge Prompting for Task Driven Object Detection](https://mlsanity.com/paper/2309.01093/) (2023)
- [Causal Reasoning Meets Visual Representation Learning: A Prospective Study](https://mlsanity.com/paper/2204.12037/) (2022)
- [Vision+X: A Survey on Multimodal Learning in the Light of Data](https://mlsanity.com/paper/2210.02884/) (2022)
- [Adapting CLIP For Phrase Localization Without Further Training](https://mlsanity.com/paper/2204.03647/) (2022)
- [SAM as the Guide: Mastering Pseudo-Label Refinement in Semi-Supervised Referring Expression Segmentation](https://mlsanity.com/paper/2406.01451/) (2024)
- [Who's Waldo? Linking People Across Text and Images](https://mlsanity.com/paper/2108.07253/) (2021)
- [Toward Explainable and Fine-Grained 3D Grounding through Referring Textual Phrases](https://mlsanity.com/paper/2207.01821/) (2022)
- [Weakly-Supervised Video Object Grounding via Causal Intervention](https://mlsanity.com/paper/2112.00475/) (2021)
- [Effectively Leveraging Attributes for Visual Similarity](https://mlsanity.com/paper/2105.01695/) (2021)
- [Weakly Supervised Human-Object Interaction Detection in Video via Contrastive Spatiotemporal Regions](https://mlsanity.com/paper/2110.03562/) (2021)
- [Adaptive Visual Scene Understanding: Incremental Scene Graph Generation](https://mlsanity.com/paper/2310.01636/) (2023)
- [2.5D Visual Relationship Detection](https://mlsanity.com/paper/2104.12727/) (2021)
- [Unpaired Referring Expression Grounding via Bidirectional Cross-Modal Matching](https://mlsanity.com/paper/2201.06686/) (2022)
- [Complex Scene Image Editing by Scene Graph Comprehension](https://mlsanity.com/paper/2203.12849/) (2022)
- [OV-VG: A Benchmark for Open-Vocabulary Visual Grounding](https://mlsanity.com/paper/2310.14374/) (2023)
- [Neural Message Passing for Visual Relationship Detection](https://mlsanity.com/paper/2208.04165/) (2022)
- [Tackling the Challenges in Scene Graph Generation with Local-to-Global Interactions](https://mlsanity.com/paper/2106.08543/) (2021)
- [Context Does Matter: End-to-end Panoptic Narrative Grounding with Deformable Attention Refined Matching Network](https://mlsanity.com/paper/2310.16616/) (2023)
- [Fine-Grained Visual Entailment](https://mlsanity.com/paper/2203.15704/) (2022)
- [Language-Guided Diffusion Model for Visual Grounding](https://mlsanity.com/paper/2308.09599/) (2023)
- [Fine-Grained Open-Vocabulary Object Detection with Fined-Grained Prompts: Task, Dataset and Benchmark](https://mlsanity.com/paper/2503.14862/) (2025)
- [Dataset Summarization by K Principal Concepts](https://mlsanity.com/paper/2104.03952/) (2021)
- [A Better Loss for Visual-Textual Grounding](https://mlsanity.com/paper/2108.05308/) (2021)

## Abstract (arXiv)

This paper presents a framework for localization or grounding of phrases in images using a large collection of linguistic and visual cues. We model the appearance, size, and position of entity bounding boxes, adjectives that contain attribute information, and spatial relationships between pairs of entities connected by verbs or prepositions. Special attention is given to relationships between people and clothing or body part mentions, as they are useful for distinguishing individuals. We automatically learn weights for combining these cues and at test time, perform joint inference over all phrases in a caption. The resulting system produces state of the art performance on phrase localization on the Flickr30k Entities dataset and visual relationship detection on the Stanford VRD dataset.