# Unpaired Referring Expression Grounding via Bidirectional Cross-Modal Matching

arXiv: https://arxiv.org/abs/2201.06686 · HTML: https://mlsanity.com/paper/2201.06686/

## Key facts

- Cited by 7 papers in the map (data through 14 Oct 2026).
- Builds on 41 of its 64 references that are in the map.
- First posted on arXiv 18 Jan 2022.

## Builds on (most cited first, 30 of 41)

- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [Microsoft COCO: Common Objects in Context](https://mlsanity.com/paper/1405.0312/) (2014)
- [Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks](https://mlsanity.com/paper/1506.01497/) (2015)
- [Grad-CAM: Visual Explanations from Deep Networks via Gradient-based Localization](https://mlsanity.com/paper/1610.02391/) (2016)
- [Visual Genome: Connecting Language and Vision Using Crowdsourced Dense Image Annotations](https://mlsanity.com/paper/1602.07332/) (2016)
- [Flickr30k Entities: Collecting Region-to-Phrase Correspondences for Richer Image-to-Sentence Models](https://mlsanity.com/paper/1505.04870/) (2015)
- [Modeling Context in Referring Expressions](https://mlsanity.com/paper/1608.00272/) (2016)
- [Generation and Comprehension of Unambiguous Object Descriptions](https://mlsanity.com/paper/1511.02283/) (2015)
- [MDETR -- Modulated Detection for End-to-End Multi-Modal Understanding](https://mlsanity.com/paper/2104.12763/) (2021)
- [MAttNet: Modular Attention Network for Referring Expression Comprehension](https://mlsanity.com/paper/1801.08186/) (2018)
- [K-Adapter: Infusing Knowledge into Pre-Trained Models with Adapters](https://mlsanity.com/paper/2002.01808/) (2020)
- [TransVG: End-to-End Visual Grounding with Transformers](https://mlsanity.com/paper/2104.08541/) (2021)
- [Modeling Context Between Objects for Referring Expression Understanding](https://mlsanity.com/paper/1608.00525/) (2016)
- [Multi-task Collaborative Network for Joint Referring Expression Comprehension and Segmentation](https://mlsanity.com/paper/2003.08813/) (2020)
- [Natural Language Object Retrieval](https://mlsanity.com/paper/1511.04164/) (2015)
- [Grounding of Textual Phrases in Images by Reconstruction](https://mlsanity.com/paper/1511.03745/) (2015)
- [Dynamic Graph Attention for Referring Expression Comprehension](https://mlsanity.com/paper/1909.08164/) (2019)
- [Contrastive Learning for Weakly Supervised Phrase Grounding](https://mlsanity.com/paper/2006.09920/) (2020)
- [Grounding Referring Expressions in Images by Variational Context](https://mlsanity.com/paper/1712.01892/) (2017)
- [A Joint Speaker-Listener-Reinforcer Model for Referring Expressions](https://mlsanity.com/paper/1612.09542/) (2016)
- [Look Before You Leap: Learning Landmark Features for One-Stage Visual Grounding](https://mlsanity.com/paper/2104.04386/) (2021)
- [Improving Weakly Supervised Visual Grounding by Contrastive Knowledge Distillation](https://mlsanity.com/paper/2007.01951/) (2020)
- [Conditional Image-Text Embedding Networks](https://mlsanity.com/paper/1711.08389/) (2017)
- [Learning Cross-modal Context Graph for Visual Grounding](https://mlsanity.com/paper/1911.09042/) (2019)
- [Weakly-supervised Visual Grounding of Phrases with Linguistic Structures](https://mlsanity.com/paper/1705.01371/) (2017)
- [Relation-aware Instance Refinement for Weakly Supervised Visual Grounding](https://mlsanity.com/paper/2103.12989/) (2021)
- [Phrase Localization and Visual Relationship Detection with Comprehensive Image-Language Cues](https://mlsanity.com/paper/1611.06641/) (2016)
- [Knowledge Aided Consistency for Weakly Supervised Phrase Grounding](https://mlsanity.com/paper/1803.03879/) (2018)
- [Fairness for Image Generation with Uncertain Sensitive Attributes](https://mlsanity.com/paper/2106.12182/) (2021)

## Built on by (most cited first, 7 of 7)

- [Toward Visual Grounding: A Survey](https://mlsanity.com/paper/2412.20206/) (2024)
- [CLIP-VG: Self-paced Curriculum Adapting of CLIP for Visual Grounding](https://mlsanity.com/paper/2305.08685/) (2023)
- [CLIP-PAE: Projection-Augmentation Embedding to Extract Relevant Features for a Disentangled, Interpretable, and Controllable Text-Guided Face Manipulation](https://mlsanity.com/paper/2210.03919/) (2022)
- [Open-Vocabulary Object Detection via Scene Graph Discovery](https://mlsanity.com/paper/2307.03339/) (2023)
- [OV-VG: A Benchmark for Open-Vocabulary Visual Grounding](https://mlsanity.com/paper/2310.14374/) (2023)
- [Unified Open-Vocabulary Dense Visual Prediction](https://mlsanity.com/paper/2307.08238/) (2023)
- [Prototype-Aware Multimodal Alignment for Open-Vocabulary Visual Grounding](https://mlsanity.com/paper/2509.06291/) (2025)

## Abstract (arXiv)

Referring expression grounding is an important and challenging task in computer vision. To avoid the laborious annotation in conventional referring grounding, unpaired referring grounding is introduced, where the training data only contains a number of images and queries without correspondences. The few existing solutions to unpaired referring grounding are still preliminary, due to the challenges of learning image-text matching and lack of the top-down guidance with unpaired data. In this paper, we propose a novel bidirectional cross-modal matching (BiCM) framework to address these challenges. Particularly, we design a query-aware attention map (QAM) module that introduces top-down perspective via generating query-specific visual attention maps. A cross-modal object matching (COM) module is further introduced, which exploits the recently emerged image-text matching pretrained model, CLIP, to predict the target objects from a bottom-up perspective. The top-down and bottom-up predictions are then integrated via a similarity funsion (SF) module. We also propose a knowledge adaptation matching (KAM) module that leverages unpaired training data to adapt pretrained knowledge to the target dataset and task. Experiments show that our framework outperforms previous works by 6.55% and 9.94% on two popular grounding datasets.