# Disentangled Motif-aware Graph Learning for Phrase Grounding

arXiv: https://arxiv.org/abs/2104.06008 · HTML: https://mlsanity.com/paper/2104.06008/

## Key facts

- Cited by 17 papers in the map (data through 14 Oct 2026).
- Builds on 32 of its 42 references that are in the map.
- First posted on arXiv 13 Apr 2021.

## Builds on (most cited first, 30 of 32)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks](https://mlsanity.com/paper/1506.01497/) (2015)
- [Graph Attention Networks](https://mlsanity.com/paper/1710.10903/) (2017)
- [Representation Learning with Contrastive Predictive Coding](https://mlsanity.com/paper/1807.03748/) (2018)
- [Mask R-CNN](https://mlsanity.com/paper/1703.06870/) (2017)
- [How Powerful are Graph Neural Networks?](https://mlsanity.com/paper/1810.00826/) (2018)
- [ViLBERT: Pretraining Task-Agnostic Visiolinguistic Representations for Vision-and-Language Tasks](https://mlsanity.com/paper/1908.02265/) (2019)
- [Flickr30k Entities: Collecting Region-to-Phrase Correspondences for Richer Image-to-Sentence Models](https://mlsanity.com/paper/1505.04870/) (2015)
- [Deep Learning and the Information Bottleneck Principle](https://mlsanity.com/paper/1503.02406/) (2015)
- [MAttNet: Modular Attention Network for Referring Expression Comprehension](https://mlsanity.com/paper/1801.08186/) (2018)
- [Neural Motifs: Scene Graph Parsing with Global Context](https://mlsanity.com/paper/1711.06640/) (2017)
- [Understanding disentangling in β-VAE](https://mlsanity.com/paper/1804.03599/) (2018)
- [Multimodal Compact Bilinear Pooling for Visual Question Answering and Visual Grounding](https://mlsanity.com/paper/1606.01847/) (2016)
- [A Fast and Accurate One-Stage Approach to Visual Grounding](https://mlsanity.com/paper/1908.06354/) (2019)
- [RUBi: Reducing Unimodal Biases in Visual Question Answering](https://mlsanity.com/paper/1906.10169/) (2019)
- [Isolating Sources of Disentanglement in Variational Autoencoders](https://mlsanity.com/paper/1802.04942/) (2018)
- [Natural Language Object Retrieval](https://mlsanity.com/paper/1511.04164/) (2015)
- [Learning Two-Branch Neural Networks for Image-Text Matching Tasks](https://mlsanity.com/paper/1704.03470/) (2017)
- [Grounding of Textual Phrases in Images by Reconstruction](https://mlsanity.com/paper/1511.03745/) (2015)
- [Visual Commonsense R-CNN](https://mlsanity.com/paper/2002.12204/) (2020)
- [Neighbourhood Watch: Referring Expression Comprehension via Language-guided Graph Attention Networks](https://mlsanity.com/paper/1812.04794/) (2018)
- [Improving Referring Expression Grounding with Cross-modal Attention-guided Erasing](https://mlsanity.com/paper/1903.00839/) (2019)
- [Zero-Shot Grounding of Objects from Natural Language Queries](https://mlsanity.com/paper/1908.07129/) (2019)
- [Comprehension-guided referring expressions](https://mlsanity.com/paper/1701.03439/) (2017)
- [Conditional Image-Text Embedding Networks](https://mlsanity.com/paper/1711.08389/) (2017)
- [Learning Cross-modal Context Graph for Visual Grounding](https://mlsanity.com/paper/1911.09042/) (2019)
- [Query-guided Regression Network with Context Policy for Phrase Grounding](https://mlsanity.com/paper/1708.01676/) (2017)
- [Multi-level Multimodal Common Semantic Space for Image-Phrase Grounding](https://mlsanity.com/paper/1811.11683/) (2018)
- [Neural Sequential Phrase Grounding (SeqGROUND)](https://mlsanity.com/paper/1903.07669/) (2019)

## Built on by (most cited first, 17 of 17)

- [Making the Most of Text Semantics to Improve Biomedical Vision--Language Processing](https://mlsanity.com/paper/2204.09817/) (2022)
- [A Survey on Graph Neural Networks and Graph Transformers in Computer Vision: A Task-Oriented Perspective](https://mlsanity.com/paper/2209.13232/) (2022)
- [Shifting More Attention to Visual Backbone: Query-modulated Refinement Networks for End-to-End Visual Grounding](https://mlsanity.com/paper/2203.15442/) (2022)
- [The devil is in the fine-grained details: Evaluating open-vocabulary object detectors for fine-grained understanding](https://mlsanity.com/paper/2311.17518/) (2023)
- [DQ-DETR: Dual Query Detection Transformer for Phrase Extraction and Grounding](https://mlsanity.com/paper/2211.15516/) (2022)
- [Adapting CLIP For Phrase Localization Without Further Training](https://mlsanity.com/paper/2204.03647/) (2022)
- [PPMN: Pixel-Phrase Matching Network for One-Stage Panoptic Narrative Grounding](https://mlsanity.com/paper/2208.05647/) (2022)
- [Robustly Optimized Deep Feature Decoupling Network for Fatty Liver Diseases Detection](https://mlsanity.com/paper/2406.17338/) (2024)
- [NICE: Improving Panoptic Narrative Detection and Segmentation with Cascading Collaborative Learning](https://mlsanity.com/paper/2310.10975/) (2023)
- [Unpaired Referring Expression Grounding via Bidirectional Cross-Modal Matching](https://mlsanity.com/paper/2201.06686/) (2022)
- [Enriching Phrases with Coupled Pixel and Object Contexts for Panoptic Narrative Grounding](https://mlsanity.com/paper/2311.01091/) (2023)
- [Context Does Matter: End-to-end Panoptic Narrative Grounding with Deformable Attention Refined Matching Network](https://mlsanity.com/paper/2310.16616/) (2023)
- [DATE: Domain Adaptive Product Seeker for E-commerce](https://mlsanity.com/paper/2304.03669/) (2023)
- [Edge-Enhanced Global Disentangled Graph Neural Network for Sequential Recommendation](https://mlsanity.com/paper/2111.10539/) (2021)
- [Extending Phrase Grounding with Pronouns in Visual Dialogues](https://mlsanity.com/paper/2210.12658/) (2022)
- [Bridging Modality Gap for Visual Grounding with Effecitve Cross-modal Distillation](https://mlsanity.com/paper/2312.17648/) (2023)
- [MotifDisco: Motif Causal Discovery For Time Series Motifs](https://mlsanity.com/paper/2409.15219/) (2024)

## Abstract (arXiv)

In this paper, we propose a novel graph learning framework for phrase grounding in the image. Developing from the sequential to the dense graph model, existing works capture coarse-grained context but fail to distinguish the diversity of context among phrases and image regions. In contrast, we pay special attention to different motifs implied in the context of the scene graph and devise the disentangled graph network to integrate the motif-aware contextual information into representations. Besides, we adopt interventional strategies at the feature and the structure levels to consolidate and generalize representations. Finally, the cross-modal attention network is utilized to fuse intra-modal features, where each phrase can be computed similarity with regions to select the best-grounded one. We validate the efficiency of disentangled and interventional graph network (DIGN) through a series of ablation studies, and our model achieves state-of-the-art performance on Flickr30K Entities and ReferIt Game benchmarks.