# BGT-Net: Bidirectional GRU Transformer Network for Scene Graph Generation

arXiv: https://arxiv.org/abs/2109.05346 · HTML: https://mlsanity.com/paper/2109.05346/

## Key facts

- Cited by 18 papers in the map (data through 14 Oct 2026).
- Builds on 36 of its 50 references that are in the map.
- First posted on arXiv 11 Sep 2021.

## Builds on (most cited first, 30 of 36)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Microsoft COCO: Common Objects in Context](https://mlsanity.com/paper/1405.0312/) (2014)
- [Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks](https://mlsanity.com/paper/1506.01497/) (2015)
- [Layer Normalization](https://mlsanity.com/paper/1607.06450/) (2016)
- [Visual Genome: Connecting Language and Vision Using Crowdsourced Dense Image Annotations](https://mlsanity.com/paper/1602.07332/) (2016)
- [Aggregated Residual Transformations for Deep Neural Networks](https://mlsanity.com/paper/1611.05431/) (2016)
- [Learning both Weights and Connections for Efficient Neural Networks](https://mlsanity.com/paper/1506.02626/) (2015)
- [Scene Graph Generation by Iterative Message Passing](https://mlsanity.com/paper/1701.02426/) (2017)
- [Neural Motifs: Scene Graph Parsing with Global Context](https://mlsanity.com/paper/1711.06640/) (2017)
- [Unbiased Scene Graph Generation from Biased Training](https://mlsanity.com/paper/2002.11949/) (2020)
- [Visual Relationship Detection with Language Priors](https://mlsanity.com/paper/1608.00187/) (2016)
- [Graph R-CNN for Scene Graph Generation](https://mlsanity.com/paper/1808.00191/) (2018)
- [Learning to Compose Dynamic Tree Structures for Visual Contexts](https://mlsanity.com/paper/1812.01880/) (2018)
- [Auto-Encoding Scene Graphs for Image Captioning](https://mlsanity.com/paper/1812.02378/) (2018)
- [GPS-Net: Graph Property Sensing Network for Scene Graph Generation](https://mlsanity.com/paper/2003.12962/) (2020)
- [Knowledge-Embedded Routing Network for Scene Graph Generation](https://mlsanity.com/paper/1903.03326/) (2019)
- [Scene Graph Generation from Objects, Phrases and Region Captions](https://mlsanity.com/paper/1707.09700/) (2017)
- [Graph-Structured Representations for Visual Question Answering](https://mlsanity.com/paper/1609.05600/) (2016)
- [Visual Translation Embedding Network for Visual Relation Detection](https://mlsanity.com/paper/1702.08319/) (2017)
- [Explainable and Explicit Visual Reasoning over Scene Graphs](https://mlsanity.com/paper/1812.01855/) (2018)
- [Graphical Contrastive Losses for Scene Graph Parsing](https://mlsanity.com/paper/1903.02728/) (2019)
- [Bridging Knowledge Graphs to Generate Scene Graphs](https://mlsanity.com/paper/2001.02314/) (2020)
- [Factorizable Net: An Efficient Subgraph-based Framework for Scene Graph Generation](https://mlsanity.com/paper/1806.11538/) (2018)
- [PCPL: Predicate-Correlation Perception Learning for Unbiased Scene Graph Generation](https://mlsanity.com/paper/2009.00893/) (2020)
- [Attentive Relational Networks for Mapping Images to Scene Graphs](https://mlsanity.com/paper/1811.10696/) (2018)
- [Visual Relationship Detection with Internal and External Linguistic Knowledge Distillation](https://mlsanity.com/paper/1707.09423/) (2017)
- [Counterfactual Critic Multi-Agent Training for Scene Graph Generation](https://mlsanity.com/paper/1812.02347/) (2018)
- [Mapping Images to Scene Graphs with Permutation-Invariant Structured Prediction](https://mlsanity.com/paper/1802.05451/) (2018)
- [Zoom-Net: Mining Deep Feature Interactions for Visual Relationship Recognition](https://mlsanity.com/paper/1807.04979/) (2018)
- [Learning to Count Objects in Natural Images for Visual Question Answering](https://mlsanity.com/paper/1802.05766/) (2018)

## Built on by (most cited first, 18 of 18)

- [Deep Long-Tailed Learning: A Survey](https://mlsanity.com/paper/2110.04596/) (2021)
- [RelTR: Relation Transformer for Scene Graph Generation](https://mlsanity.com/paper/2201.11460/) (2022)
- [Scene Graph Generation: A Comprehensive Survey](https://mlsanity.com/paper/2201.00443/) (2022)
- [Relationformer: A Unified Framework for Image-to-Graph Generation](https://mlsanity.com/paper/2203.10202/) (2022)
- [EGTR: Extracting Graph from Transformer for Scene Graph Generation](https://mlsanity.com/paper/2404.02072/) (2024)
- [Graph Transformers: A Survey](https://mlsanity.com/paper/2407.09777/) (2024)
- [Hierarchical Memory Learning for Fine-Grained Scene Graph Generation](https://mlsanity.com/paper/2203.06907/) (2022)
- [Adaptive Fine-Grained Predicates Learning for Scene Graph Generation](https://mlsanity.com/paper/2207.04602/) (2022)
- [Reusing Attention for One-stage Lane Topology Understanding](https://mlsanity.com/paper/2507.17617/) (2025)
- [SSGVS: Semantic Scene Graph-to-Video Synthesis](https://mlsanity.com/paper/2211.06119/) (2022)
- [Enhancing Scene Graph Generation with Hierarchical Relationships and Commonsense Knowledge](https://mlsanity.com/paper/2311.12889/) (2023)
- [DAVINCI: A Single-Stage Architecture for Constrained CAD Sketch Inference](https://mlsanity.com/paper/2410.22857/) (2024)
- [Explore Contextual Information for 3D Scene Graph Generation](https://mlsanity.com/paper/2210.06240/) (2022)
- [Hierarchical Relationships: A New Perspective to Enhance Scene Graph Generation](https://mlsanity.com/paper/2303.06842/) (2023)
- [SPAN: Learning Similarity between Scene Graphs and Images with Transformers](https://mlsanity.com/paper/2304.00590/) (2023)
- [RepSGG: Novel Representations of Entities and Relationships for Scene Graph Generation](https://mlsanity.com/paper/2309.03240/) (2023)
- [S^2Former-OR: Single-Stage Bi-Modal Transformer for Scene Graph Generation in OR](https://mlsanity.com/paper/2402.14461/) (2024)
- [SGFormer++: Semantic Graph Transformer for Incremental 3D Scene Graph Generation](https://mlsanity.com/paper/2606.15328/) (2026)

## Abstract (arXiv)

Scene graphs are nodes and edges consisting of objects and object-object relationships, respectively. Scene graph generation (SGG) aims to identify the objects and their relationships. We propose a bidirectional GRU (BiGRU) transformer network (BGT-Net) for the scene graph generation for images. This model implements novel object-object communication to enhance the object information using a BiGRU layer. Thus, the information of all objects in the image is available for the other objects, which can be leveraged later in the object prediction step. This object information is used in a transformer encoder to predict the object class as well as to create object-specific edge information via the use of another transformer encoder. To handle the dataset bias induced by the long-tailed relationship distribution, softening with a log-softmax function and adding a bias adaptation term to regulate the bias for every relation prediction individually showed to be an effective approach. We conducted an elaborate study on experiments and ablations using open-source datasets, i.e., Visual Genome, Open-Images, and Visual Relationship Detection datasets, demonstrating the effectiveness of the proposed model over state of the art.