# Referring Relationships

arXiv: https://arxiv.org/abs/1803.10362 · HTML: https://mlsanity.com/paper/1803.10362/

## Key facts

- Cited by 29 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 28 Mar 2018.

## Built on by (most cited first, 29 of 29)

- [A Comprehensive Survey of Scene Graphs: Generation and Application](https://mlsanity.com/paper/2104.01111/) (2021)
- [AGQA: A Benchmark for Compositional Spatio-Temporal Reasoning](https://mlsanity.com/paper/2103.16002/) (2021)
- [Object-Region Video Transformers](https://mlsanity.com/paper/2110.06915/) (2021)
- [Scene Graph Generation: A Comprehensive Survey](https://mlsanity.com/paper/2201.00443/) (2022)
- [Dense and Aligned Captions (DAC) Promote Compositional Reasoning in VL Models](https://mlsanity.com/paper/2305.19595/) (2023)
- [Teaching Structured Vision&Language Concepts to Vision&Language Models](https://mlsanity.com/paper/2211.11733/) (2022)
- [Video Graph Transformer for Video Question Answering](https://mlsanity.com/paper/2207.05342/) (2022)
- [Incorporating Structured Representations into Pretrained Vision & Language Models Using Scene Graphs](https://mlsanity.com/paper/2305.06343/) (2023)
- [Contrastive Video Question Answering via Video Graph Transformer](https://mlsanity.com/paper/2302.13668/) (2023)
- [Helping Hands: An Object-Aware Ego-Centric Video Recognition Model](https://mlsanity.com/paper/2308.07918/) (2023)
- [TraveLER: A Modular Multi-LMM Agent Framework for Video Question-Answering](https://mlsanity.com/paper/2404.01476/) (2024)
- [Is an Object-Centric Video Representation Beneficial for Transfer?](https://mlsanity.com/paper/2207.10075/) (2022)
- [Zero-shot Referring Expression Comprehension via Structural Similarity Between Images and Captions](https://mlsanity.com/paper/2311.17048/) (2023)
- [Vision Relation Transformer for Unbiased Scene Graph Generation](https://mlsanity.com/paper/2308.09472/) (2023)
- [SGEITL: Scene Graph Enhanced Image-Text Learning for Visual Commonsense Reasoning](https://mlsanity.com/paper/2112.08587/) (2021)
- [PromptonomyViT: Multi-Task Prompt Learning Improves Video Transformers using Synthetic Scene Data](https://mlsanity.com/paper/2212.04821/) (2022)
- [Diagnostic Benchmark and Iterative Inpainting for Layout-Guided Image Generation](https://mlsanity.com/paper/2304.06671/) (2023)
- [Complex Scene Image Editing by Scene Graph Comprehension](https://mlsanity.com/paper/2203.12849/) (2022)
- [3VL: Using Trees to Improve Vision-Language Models' Interpretability](https://mlsanity.com/paper/2312.17345/) (2023)
- [AUG: A New Dataset and An Efficient Model for Aerial Image Urban Scene Graph Generation](https://mlsanity.com/paper/2404.07788/) (2024)
- [PatchGame: Learning to Signal Mid-level Patches in Referential Games](https://mlsanity.com/paper/2111.01785/) (2021)
- [DetermiNet: A Large-Scale Diagnostic Dataset for Complex Visually-Grounded Referencing using Determiners](https://mlsanity.com/paper/2309.03483/) (2023)
- [Towards Flexible Visual Relationship Segmentation](https://mlsanity.com/paper/2408.08305/) (2024)
- [Image interpretation by iterative bottom-up top-down processing](https://mlsanity.com/paper/2105.05592/) (2021)
- [Grounding Scene Graphs on Natural Images via Visio-Lingual Message Passing](https://mlsanity.com/paper/2211.01969/) (2022)
- [NAVERO: Unlocking Fine-Grained Semantics for Video-Language Compositionality](https://mlsanity.com/paper/2408.09511/) (2024)
- [Few-shot Visual Relationship Co-localization](https://mlsanity.com/paper/2108.11618/) (2021)
- [Bringing Image Scene Structure to Video via Frame-Clip Consistency of Object Tokens](https://mlsanity.com/paper/2206.06346/) (2022)
- [Vision-Language Grounding as Bidirectional Concept Correspondence](https://mlsanity.com/paper/2608.07886/) (2026)

## Abstract (arXiv)

Images are not simply sets of objects: each image represents a web of interconnected relationships. These relationships between entities carry semantic meaning and help a viewer differentiate between instances of an entity. For example, in an image of a soccer match, there may be multiple persons present, but each participates in different relationships: one is kicking the ball, and the other is guarding the goal. In this paper, we formulate the task of utilizing these "referring relationships" to disambiguate between entities of the same category. We introduce an iterative model that localizes the two entities in the referring relationship, conditioned on one another. We formulate the cyclic condition between the entities in a relationship by modelling predicates that connect the entities as shifts in attention from one entity to another. We demonstrate that our model can not only outperform existing approaches on three datasets --- CLEVR, VRD and Visual Genome --- but also that it produces visually meaningful predicate shifts, as an instance of interpretable neural networks. Finally, we show that by modelling predicates as attention shifts, we can even localize entities in the absence of their category, allowing our model to find completely unseen categories.