# RelationNet++: Bridging Visual Representations for Object Detection via Transformer Decoder

arXiv: https://arxiv.org/abs/2010.15831 · HTML: https://mlsanity.com/paper/2010.15831/

## Key facts

- Cited by 31 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 29 Oct 2020.

## Built on by (most cited first, 30 of 31)

- [Swin Transformer: Hierarchical Vision Transformer using Shifted Windows](https://mlsanity.com/paper/2103.14030/) (2021)
- [AdaptFormer: Adapting Vision Transformers for Scalable Visual Recognition](https://mlsanity.com/paper/2205.13535/) (2022)
- [Learning to Prompt for Open-Vocabulary Object Detection with Vision-Language Model](https://mlsanity.com/paper/2203.14940/) (2022)
- [CMT: Convolutional Neural Networks Meet Vision Transformers](https://mlsanity.com/paper/2107.06263/) (2021)
- [Group-Free 3D Object Detection via Transformers](https://mlsanity.com/paper/2104.00678/) (2021)
- [Dynamic Head: Unifying Object Detection Heads with Attentions](https://mlsanity.com/paper/2106.08322/) (2021)
- [Semi-Supervised 3D Hand-Object Poses Estimation with Interactions in Time](https://mlsanity.com/paper/2106.05266/) (2021)
- [MGFN: Magnitude-Contrastive Glance-and-Focus Network for Weakly-Supervised Video Anomaly Detection](https://mlsanity.com/paper/2211.15098/) (2022)
- [Gaze Estimation using Transformer](https://mlsanity.com/paper/2105.14424/) (2021)
- [A Survey on Graph Neural Networks and Graph Transformers in Computer Vision: A Task-Oriented Perspective](https://mlsanity.com/paper/2209.13232/) (2022)
- [CP-ViT: Cascade Vision Transformer Pruning via Progressive Sparsity Prediction](https://mlsanity.com/paper/2203.04570/) (2022)
- [Revitalizing CNN Attentions via Transformers in Self-Supervised Visual Representation Learning](https://mlsanity.com/paper/2110.05340/) (2021)
- [BigDetection: A Large-scale Benchmark for Improved Object Detector Pre-training](https://mlsanity.com/paper/2203.13249/) (2022)
- [Cross-Layer Retrospective Retrieving via Layer Attention](https://mlsanity.com/paper/2302.03985/) (2023)
- [Transformers in Small Object Detection: A Benchmark and Survey of State-of-the-Art](https://mlsanity.com/paper/2309.04902/) (2023)
- [SARAS-Net: Scale and Relation Aware Siamese Network for Change Detection](https://mlsanity.com/paper/2212.01287/) (2022)
- [Rotation Invariant Transformer for Recognizing Object in UAVs](https://mlsanity.com/paper/2311.02559/) (2023)
- [USB: Universal-Scale Object Detection Benchmark](https://mlsanity.com/paper/2103.14027/) (2021)
- [Oriented object detection in optical remote sensing images using deep learning: a survey](https://mlsanity.com/paper/2302.10473/) (2023)
- [Transformers Meet Visual Learning Understanding: A Comprehensive Review](https://mlsanity.com/paper/2203.12944/) (2022)
- [R(Det)^2: Randomized Decision Routing for Object Detection](https://mlsanity.com/paper/2204.00794/) (2022)
- [FQDet: Fast-converging Query-based Detector](https://mlsanity.com/paper/2210.02318/) (2022)
- [HoughNet: Integrating near and long-range evidence for visual detection](https://mlsanity.com/paper/2104.06773/) (2021)
- [UniInst: Unique Representation for End-to-End Instance Segmentation](https://mlsanity.com/paper/2205.12646/) (2022)
- [A More Compact Object Detector Head Network with Feature Enhancement and Relational Reasoning](https://mlsanity.com/paper/2106.14475/) (2021)
- [SEFormer: Structure Embedding Transformer for 3D Object Detection](https://mlsanity.com/paper/2209.01745/) (2022)
- [Degenerate Swin to Win: Plain Window-based Transformer without Sophisticated Operations](https://mlsanity.com/paper/2211.14255/) (2022)
- [RelationRS: Relationship Representation Network for Object Detection in Aerial Images](https://mlsanity.com/paper/2110.06730/) (2021)
- [HR-RCNN: Hierarchical Relational Reasoning for Object Detection](https://mlsanity.com/paper/2110.13892/) (2021)
- [Future Object Detection with Spatiotemporal Transformers](https://mlsanity.com/paper/2204.10321/) (2022)

## Abstract (arXiv)

Existing object detection frameworks are usually built on a single format of object/part representation, i.e., anchor/proposal rectangle boxes in RetinaNet and Faster R-CNN, center points in FCOS and RepPoints, and corner points in CornerNet. While these different representations usually drive the frameworks to perform well in different aspects, e.g., better classification or finer localization, it is in general difficult to combine these representations in a single framework to make good use of each strength, due to the heterogeneous or non-grid feature extraction by different representations. This paper presents an attention-based decoder module similar as that in Transformer~citevaswani2017attention to bridge other representations into a typical object detector built on a single representation format, in an end-to-end fashion. The other representations act as a set of key instances to strengthen the main query representation features in the vanilla detectors. Novel techniques are proposed towards efficient computation of the decoder module, including a key sampling approach and a shared location embedding approach. The proposed module is named bridging visual representations (BVR). It can perform in-place and we demonstrate its broad effectiveness in bridging other representations into prevalent object detection frameworks, including RetinaNet, Faster R-CNN, FCOS and ATSS, where about 1.5~3.0 AP improvements are achieved. In particular, we improve a state-of-the-art framework with a strong backbone by about 2.0 AP, reaching 52.7 AP on COCO test-dev. The resulting network is named RelationNet++. The code will be available at https://github.com/microsoft/RelationNet2.