# Detecting Visual Relationships Using Box Attention

arXiv: https://arxiv.org/abs/1807.02136 · HTML: https://mlsanity.com/paper/1807.02136/

## Key facts

- Cited by 16 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 5 Jul 2018.

## Built on by (most cited first, 16 of 16)

- [HOTR: End-to-End Human-Object Interaction Detection with Transformers](https://mlsanity.com/paper/2104.13682/) (2021)
- [Rendezvous: Attention Mechanisms for the Recognition of Surgical Action Triplets in Endoscopic Videos](https://mlsanity.com/paper/2109.03223/) (2021)
- [A Comprehensive Survey of Scene Graphs: Generation and Application](https://mlsanity.com/paper/2104.01111/) (2021)
- [Panoptic Scene Graph Generation](https://mlsanity.com/paper/2207.11247/) (2022)
- [UViM: A Unified Modeling Approach for Vision with Learned Guiding Codes](https://mlsanity.com/paper/2205.10337/) (2022)
- [Scene Graph Generation: A Comprehensive Survey](https://mlsanity.com/paper/2201.00443/) (2022)
- [VINS: Visual Search for Mobile User Interface Design](https://mlsanity.com/paper/2102.05216/) (2021)
- [MSTR: Multi-Scale Transformer for End-to-End Human-Object Interaction Detection](https://mlsanity.com/paper/2203.14709/) (2022)
- [UnionDet: Union-Level Detector Towards Real-Time Human-Object Interaction Detection](https://mlsanity.com/paper/2312.12664/) (2023)
- [Consistency Learning via Decoding Path Augmentation for Transformers in Human Object Interaction Detection](https://mlsanity.com/paper/2204.04836/) (2022)
- [Cross-Modality Time-Variant Relation Learning for Generating Dynamic Scene Graphs](https://mlsanity.com/paper/2305.08522/) (2023)
- [Towards Lifelong Scene Graph Generation with Knowledge-ware In-context Prompt Learning](https://mlsanity.com/paper/2401.14626/) (2024)
- [HAtt-Flow: Hierarchical Attention-Flow Mechanism for Group Activity Scene Graph Generation in Videos](https://mlsanity.com/paper/2312.07740/) (2023)
- [Human Object Interaction Detection using Two-Direction Spatial Enhancement and Exclusive Object Prior](https://mlsanity.com/paper/2105.03089/) (2021)
- [SGDraw: Scene Graph Drawing Interface Using Object-Oriented Representation](https://mlsanity.com/paper/2211.16697/) (2022)
- [UI Layers Group Detector: Grouping UI Layers via Text Fusion and Box Attention](https://mlsanity.com/paper/2212.03440/) (2022)

## Abstract (arXiv)

We propose a new model for detecting visual relationships, such as "person riding motorcycle" or "bottle on table". This task is an important step towards comprehensive structured image understanding, going beyond detecting individual objects. Our main novelty is a Box Attention mechanism that allows to model pairwise interactions between objects using standard object detection pipelines. The resulting model is conceptually clean, expressive and relies on well-justified training and prediction procedures. Moreover, unlike previously proposed approaches, our model does not introduce any additional complex components or hyperparameters on top of those already required by the underlying detection model. We conduct an experimental evaluation on three challenging datasets, V-COCO, Visual Relationships and Open Images, demonstrating strong quantitative and qualitative results.