Data to 7 Oct 2026

Scene Graph Generation from Objects, Phrases and Region Captions

arXiv 1707.09700 ↗First posted 31 Jul 2017cs.CVcited by 154

Not on the map. Why?

Abstract (arXiv)

Object detection, scene graph generation and region captioning, which are three scene understanding tasks at different semantic levels, are tied together: scene graphs are generated on top of objects detected in an image with their pairwise relationship predicted, while region captioning gives a language description of the objects, their attributes, relations, and other context information. In this work, to leverage the mutual connections across semantic levels, we propose a novel neural network model, termed as Multi-level Scene Description Network (denoted as MSDN), to solve the three vision tasks jointly in an end-to-end manner. Objects, phrases, and caption regions are first aligned with a dynamic graph based on their spatial and semantic connections. Then a feature refining structure is used to pass messages across the three levels of semantic tasks through the graph. We benchmark the learned model on three tasks, and show the joint learning across three tasks with our proposed method can bring mutual improvements over previous models. Particularly, on the scene graph generation task, our proposed method outperforms the state-of-art method with more than 3% margin.

Key facts. Cited by 154 papers in the map (data through 7 Oct 2026). First posted on arXiv 31 Jul 2017.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 38; 2022: 45; 2023: 32; 2024: 17; 2025: 14; 2026: 8.

154 papers.

Sort:

By area:

2023GPT4RoI: Instruction Tuning Large Language Model on Region-of-Interestcited by 300
2021SceneGraphFusion: Incremental 3D Scene Graph Prediction from RGB-D Sequencescited by 159
2021Kimera: from SLAM to Spatial Perception with 3D Dynamic Scene Graphscited by 144
2021Bipartite Graph Network with Adaptive Message Passing for Unbiased Scene Graph Generationcited by 137
2021A Comprehensive Survey of Scene Graphs: Generation and Applicationcited by 125
2021SGTR: End-to-end Scene Graph Generation with Transformercited by 102
2021Spatial-Temporal Transformer for Dynamic Scene Graph Generationcited by 95
2022RelTR: Relation Transformer for Scene Graph Generationcited by 95
2024The All-Seeing Project V2: Towards General Relation Comprehension of the Open Worldcited by 90
2021Chest ImaGenome Dataset for Clinical Reasoningcited by 89
2021Graph-to-3D: End-to-End Generation and Manipulation of 3D Scenes Using Scene Graphscited by 83
2022Scene Graph Generation: A Comprehensive Surveycited by 73

38 more

2021Learning to Generate Scene Graph from Natural Language Supervisioncited by 64
2022The Devil is in the Labels: Noisy Label Correction for Robust Scene Graph Generationcited by 60
2021Fully Convolutional Scene Graph Generationcited by 57
2022Fine-Grained Scene Graph Generation with Data Transfercited by 57
2021Semantics for Robotic Mapping, Perception and Interaction: A Surveycited by 56
2021Hierarchical Representations and Explicit Memory: Learning Effective Navigation Policies on 3D Scene Graphs using Graph Neural Networkscited by 50
2021Exploiting Edge-Oriented Reasoning for 3D Point-based Scene Graph Analysiscited by 47
2022Graph Inverse Reinforcement Learning from Diverse Videoscited by 45
2022Deep Learning Approaches on Image Captioning: A Reviewcited by 43
2023SceneGenie: Scene Graph Guided Diffusion Models for Image Synthesiscited by 41
2022Continuous Scene Representations for Embodied AIcited by 39
2021Unified Graph Structured Models for Video Understandingcited by 38
2023Unbiased Scene Graph Generation in Videoscited by 38
2021Probabilistic Modeling of Semantic Ambiguity for Scene Graph Generationcited by 35
2021Multimodal Co-learning: Challenges, Applications with Datasets, Recent Advances and Future Directionscited by 35
2021A Peek Into the Reasoning of Neural Networks: Interpreting with Structural Visual Conceptscited by 33
2023Coarse-to-Fine Contrastive Learning in Image-Text-Graph Space for Improved Vision-Language Compositionalitycited by 32
2021Exploiting Long-Term Dependencies for Generating Dynamic Scene Graphscited by 30
2022End-to-End Zero-Shot HOI Detection via Vision and Language Knowledge Distillationcited by 30
2021Linguistic Structures as Weak Supervision for Visual Scene Graph Generationcited by 29
2023SGFormer: Semantic Graph Transformer for Point Cloud-based 3D Scene Graph Generationcited by 27
2021Visual Distant Supervision for Scene Graph Generationcited by 25
2022Reasoning with Scene Graphs for Robot Planning under Partial Observabilitycited by 25
2022Beyond a Pre-Trained Object Detector: Cross-Modal Textual and Visual Context for Image Captioningcited by 25
2022Iterative Scene Graph Generationcited by 24
2023Devil's on the Edges: Selective Quad Attention for Scene Graph Generationcited by 23
2022Vision+X: A Survey on Multimodal Learning in the Light of Datacited by 22
2023Category-Level Multi-Part Multi-Joint 3D Shape Assemblycited by 22
2024OED: Towards One-stage End-to-End Dynamic Scene Graph Generationcited by 22
2021Not All Relations are Equal: Mining Informative Labels for Scene Graph Generationcited by 21
2023Zero-shot Referring Expression Comprehension via Structural Similarity Between Images and Captionscited by 21
2022Integrating Object-aware and Interaction-aware Knowledge for Weakly Supervised Scene Graph Generationcited by 20
2022GSRFormer: Grounded Situation Recognition Transformer with Alternate Semantic Attention Refinementcited by 20
2023SGRec3D: Self-Supervised 3D Scene Graph Learning via Object-Level Scene Reconstructioncited by 20
2022NICEST: Noisy Label Correction and Training for Robust Scene Graph Generationcited by 19
2023HiLo: Exploiting High Low Frequency Relations for Unbiased Panoptic Scene Graph Generationcited by 19
2021BGT-Net: Bidirectional GRU Transformer Network for Scene Graph Generationcited by 18
2023Vision Relation Transformer for Unbiased Scene Graph Generationcited by 18

50 most cited of 154 All 154 (JSON)