# Knowledge-guided Pairwise Reconstruction Network for Weakly Supervised Referring Expression Grounding

arXiv: https://arxiv.org/abs/1909.02860 · HTML: https://mlsanity.com/paper/1909.02860/

## Key facts

- Cited by 21 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 5 Sep 2019.

## Built on by (most cited first, 21 of 21)

- [GroupViT: Semantic Segmentation Emerges from Text Supervision](https://mlsanity.com/paper/2202.11094/) (2022)
- [Toward Visual Grounding: A Survey](https://mlsanity.com/paper/2412.20206/) (2024)
- [Relation-aware Instance Refinement for Weakly Supervised Visual Grounding](https://mlsanity.com/paper/2103.12989/) (2021)
- [CLIP-VG: Self-paced Curriculum Adapting of CLIP for Visual Grounding](https://mlsanity.com/paper/2305.08685/) (2023)
- [Pseudo-Q: Generating Pseudo Language Queries for Visual Grounding](https://mlsanity.com/paper/2203.08481/) (2022)
- [Entity-enhanced Adaptive Reconstruction Network for Weakly Supervised Referring Expression Grounding](https://mlsanity.com/paper/2207.08386/) (2022)
- [Discriminative Triad Matching and Reconstruction for Weakly Referring Expression Grounding](https://mlsanity.com/paper/2106.04053/) (2021)
- [GroundVLP: Harnessing Zero-shot Visual Grounding from Vision-Language Pre-training and Open-Vocabulary Object Detection](https://mlsanity.com/paper/2312.15043/) (2023)
- [Efficient Vision-Language Pretraining with Visual Concepts and Hierarchical Alignment](https://mlsanity.com/paper/2208.13628/) (2022)
- [Weakly-Supervised Video Object Grounding via Causal Intervention](https://mlsanity.com/paper/2112.00475/) (2021)
- [Kefa: A Knowledge Enhanced and Fine-grained Aligned Speaker for Navigation Instruction Generation](https://mlsanity.com/paper/2307.13368/) (2023)
- [WeakMCN: Multi-task Collaborative Network for Weakly Supervised Referring Expression Comprehension and Segmentation](https://mlsanity.com/paper/2505.18686/) (2025)
- [Unpaired Referring Expression Grounding via Bidirectional Cross-Modal Matching](https://mlsanity.com/paper/2201.06686/) (2022)
- [Who are you referring to? Coreference resolution in image narrations](https://mlsanity.com/paper/2211.14563/) (2022)
- [Top-Down Framework for Weakly-supervised Grounded Image Captioning](https://mlsanity.com/paper/2306.07490/) (2023)
- [Seeing the Trees for the Forest: Rethinking Weakly-Supervised Medical Visual Grounding](https://mlsanity.com/paper/2505.15123/) (2025)
- [Advancing Visual Large Language Model for Multi-granular Versatile Perception](https://mlsanity.com/paper/2507.16213/) (2025)
- [Distributed Attention for Grounded Image Captioning](https://mlsanity.com/paper/2108.01056/) (2021)
- [AlignCAT: Visual-Linguistic Alignment of Category and Attribute for Weakly Supervised Visual Grounding](https://mlsanity.com/paper/2508.03201/) (2025)
- [LIHE: Linguistic Instance-Split Hyperbolic-Euclidean Framework for Generalized Weakly-Supervised Referring Expression Comprehension](https://mlsanity.com/paper/2511.12020/) (2025)
- [How to Understand "Support"? An Implicit-enhanced Causal Inference Approach for Weakly-supervised Phrase Grounding](https://mlsanity.com/paper/2402.19116/) (2024)

## Abstract (arXiv)

Weakly supervised referring expression grounding (REG) aims at localizing the referential entity in an image according to linguistic query, where the mapping between the image region (proposal) and the query is unknown in the training stage. In referring expressions, people usually describe a target entity in terms of its relationship with other contextual entities as well as visual attributes. However, previous weakly supervised REG methods rarely pay attention to the relationship between the entities. In this paper, we propose a knowledge-guided pairwise reconstruction network (KPRN), which models the relationship between the target entity (subject) and contextual entity (object) as well as grounds these two entities. Specifically, we first design a knowledge extraction module to guide the proposal selection of subject and object. The prior knowledge is obtained in a specific form of semantic similarities between each proposal and the subject/object. Second, guided by such knowledge, we design the subject and object attention module to construct the subject-object proposal pairs. The subject attention excludes the unrelated proposals from the candidate proposals. The object attention selects the most suitable proposal as the contextual proposal. Third, we introduce a pairwise attention and an adaptive weighting scheme to learn the correspondence between these proposal pairs and the query. Finally, a pairwise reconstruction module is used to measure the grounding for weakly supervised learning. Extensive experiments on four large-scale datasets show our method outperforms existing state-of-the-art methods by a large margin.