mlsanityData to 14 Oct 2026

Expanding Scene Graph Boundaries: Fully Open-vocabulary Scene Graph Generation via Visual-Concept Alignment and Retention

arXiv 2311.10988 ↗First posted 18 Nov 2023cs.CVcited by 29

Not on the map. Why?

Abstract (arXiv)

Scene Graph Generation (SGG) offers a structured representation critical in many computer vision applications. Traditional SGG approaches, however, are limited by a closed-set assumption, restricting their ability to recognize only predefined object and relation categories. To overcome this, we categorize SGG scenarios into four distinct settings based on the node and edge: Closed-set SGG, Open Vocabulary (object) Detection-based SGG (OvD-SGG), Open Vocabulary Relation-based SGG (OvR-SGG), and Open Vocabulary Detection + Relationbased SGG (OvD+R-SGG). While object-centric open vocabulary SGG has been studied recently, the more challenging problem of relation-involved open-vocabulary SGG remains relatively unexplored. To fill this gap, we propose a unified framework named OvSGTR towards fully open vocabulary SGG from a holistic view. The proposed framework is an end-to-end transformer architecture, which learns a visual-concept alignment for both nodes and edges, enabling the model to recognize unseen categories. For the more challenging settings of relation-involved open vocabulary SGG, the proposed approach integrates relation-aware pretraining utilizing image-caption data and retains visual-concept alignment through knowledge distillation. Comprehensive experimental results on the Visual Genome benchmark demonstrate the effectiveness and superiority of the proposed framework. Our code is available at https://github.com/gpt4vision/OvSGTR/.

Key facts. Cited by 29 papers in the map (data through 14 Oct 2026). Builds on 43 of its 53 references that are in the map. First posted on arXiv 18 Nov 2023.

Builds on

43 of 53 references.

References · 43

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  2. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  3. 2021Swin Transformer: Hierarchical Vision Transformer using Shifted Windowscited by 12,139
  4. 2015Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networkscited by 9,639
  5. 2020Deformable DETR: Deformable Transformers for End-to-End Object Detectioncited by 3,422
  6. 2023Grounding DINO: Marrying DINO with Grounded Pre-Training for Open-Set Object Detectioncited by 3,332
  7. 2015Microsoft COCO Captions: Data Collection and Evaluation Servercited by 1,462
  8. 2019Generalized Intersection over Union: A Metric and A Loss for Bounding Box Regressioncited by 1,368
  9. 2015Flickr30k Entities: Collecting Region-to-Phrase Correspondences for Richer Image-to-Sentence Modelscited by 1,362
  10. 2021Grounded Language-Image Pre-trainingcited by 1,156
  11. 2021RegionCLIP: Region-based Language-Image Pretrainingcited by 616
  12. 2021Scaling Open-Vocabulary Image Segmentation with Image-Level Labelscited by 473
31 more
  1. 2017Scene Graph Generation by Iterative Message Passingcited by 462
  2. 2020Open-Vocabulary Object Detection Using Captionscited by 427
  3. 2017Neural Motifs: Scene Graph Parsing with Global Contextcited by 414
  4. 2020Unbiased Scene Graph Generation from Biased Trainingcited by 356
  5. 2018Image Generation from Scene Graphscited by 324
  6. 2021ActionCLIP: A New Paradigm for Video Action Recognitioncited by 320
  7. 2022Learning to Prompt for Open-Vocabulary Object Detection with Vision-Language Modelcited by 317
  8. 2018Learning to Compose Dynamic Tree Structures for Visual Contextscited by 259
  9. 2018Auto-Encoding Scene Graphs for Image Captioningcited by 255
  10. 2023Towards Open Vocabulary Learning: A Surveycited by 166
  11. 2019Knowledge-Embedded Routing Network for Scene Graph Generationcited by 164
  12. 2021Bipartite Graph Network with Adaptive Message Passing for Unbiased Scene Graph Generationcited by 137
  13. 2016Graph-Structured Representations for Visual Question Answeringcited by 123
  14. 2023Aligning Bag of Regions for Open-Vocabulary Object Detectioncited by 119
  15. 2017Visual Translation Embedding Network for Visual Relation Detectioncited by 112
  16. 2021Energy-Based Learning for Scene Graph Generationcited by 104
  17. 2021SGTR: End-to-end Scene Graph Generation with Transformercited by 103
  18. 2019Graphical Contrastive Losses for Scene Graph Parsingcited by 100
  19. 2020Say As You Wish: Fine-grained Control of Image Caption Generation with Abstract Scene Graphscited by 87
  20. 2019Unpaired Image Captioning via Scene Graph Alignmentscited by 82
  21. 2021Recovering the Unbiased Scene Graphs from the Biased Onescited by 71
  22. 2021Learning to Generate Scene Graph from Natural Language Supervisioncited by 64
  23. 2023A Survey on Open-Vocabulary Detection and Segmentation: Past, Present, and Futurecited by 61
  24. 2021Fully Convolutional Scene Graph Generationcited by 57
  25. 2022Diffusion-Based Scene Graph to Image Generation with Masked Contrastive Pre-Trainingcited by 56
  26. 2022Towards Open-vocabulary Scene Graph Generation with Prompt-based Finetuningcited by 54
  27. 2021In Defense of Scene Graphs for Image Captioningcited by 31
  28. 2021Linguistic Structures as Weak Supervision for Visual Scene Graph Generationcited by 29
  29. 2022HL-Net: Heterophily Learning Network for Scene Graph Generationcited by 22
  30. 2022Integrating Object-aware and Interaction-aware Knowledge for Weakly Supervised Scene Graph Generationcited by 20
  31. 2020Generative Compositional Augmentations for Scene Graph Predictioncited by 14

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 1; 2024: 5; 2025: 12; 2026: 11.

29 papers.

Sort:

By area:

  1. 2024OpenPSG: Open-set Panoptic Scene Graph Generation via Large Multimodal Modelscited by 16
  2. 2024Scene Graph Generation with Role-Playing Large Language Modelscited by 16
  3. 2026Graph is a Substrate Across Data Modalitiescited by 14
  4. 2025Compile Scene Graphs with Reinforcement Learningcited by 11
  5. 2024Towards Scene Graph Anticipationcited by 10
  6. 2023GPT4SGG: Synthesizing Scene Graphs from Holistic and Region-specific Narrativescited by 7
  7. 2025Open World Scene Graph Generation using Vision Language Modelscited by 7
  8. 2024Towards Unbiased and Robust Spatio-Temporal Scene Graph Generation and Anticipationcited by 6
  9. 2024What Makes a Scene ? Scene Graph-based Evaluation and Feedback for Controllable Generationcited by 6
  10. 2025Learning 4D Panoptic Scene Graph Generation from Rich 2D Visual Scenecited by 6
  11. 2025Robo-SGG: Exploiting Layout-Oriented Normalization and Restitution Can Improve Robust Scene Graph Generationcited by 3
  12. 2025From Data to Modeling: Fully Open-vocabulary Scene Graph Generationcited by 2
17 more
  1. 2025Object-Centric Framework for Video Moment Retrievalcited by 2
  2. 2026The Semantic Lifecycle in Embodied AI: Acquisition, Representation and Storage via Foundation Modelscited by 2
  3. 2025FORESCENE: FOREcasting human activity via latent SCENE graphs diffusioncited by 1
  4. 2025Measuring Image-Relation Alignment: Reference-Free Evaluation of VLMs and Synthetic Pre-training for Open-Vocabulary Scene Graph Generationcited by 1
  5. 2025Spatiotemporal Knowledge Graphs as Persistent Scene Memory for Embodied Question Answeringcited by 1
  6. 2025ZING-3D: Zero-shot Incremental 3D Scene Graphs via Vision-Language Modelscited by 1
  7. 2025Hallucinate, Ground, Repeat: A Framework for Generalized Visual Relationship Detectioncited by 0
  8. 2025Interaction-Centric Knowledge Infusion and Transfer for Open-Vocabulary Scene Graph Generationcited by 0
  9. 2026SGG-Rrm 3: From Next-Token Prediction to End-to-End Unbiased Scene Graph Generationcited by 0
  10. 2026SR-Nav: Spatial Relationships Matter for Zero-shot Object Goal Navigationcited by 0
  11. 2026Caption Generation for Dongba Paintings via Prompt Learning and Semantic Fusioncited by 0
  12. 2026Can We Build Scene Graphs, Not Classify Them? FlowSG: Progressive Image-Conditioned Scene Graph Generation with Flow Matchingcited by 0
  13. 2026CAGE-SGG: Counterfactual Active Graph Evidence for Open-Vocabulary Scene Graph Generationcited by 0
  14. 2026ReLIC-SGG: Relation Lattice Completion for Open-Vocabulary Scene Graph Generationcited by 0
  15. 2026SceneGraphGrounder: Zero-Shot 3D Visual Grounding via Structured Scene Graph Matchingcited by 0
  16. 2026Modeling Scientific Experiment Scenes: Dataset and Modelcited by 0
  17. 2026OvDSGG: End-to-End Open-Vocabulary Dynamic Scene Graph Generationcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024From Pixels to Graphs: Open-Vocabulary Scene Graph Generation with Vision-Language Models
  2. 2024LLaVA-SpaceSGG: Visual Instruct Tuning for Open-vocabulary Scene Graph Generation with Enhanced Spatial Relations
  3. 2024Relation-aware Hierarchical Prompt for Open-vocabulary Scene Graph Generation
  4. 2025Taking A Closer Look at Interacting Objects: Interaction-Aware Open Vocabulary Scene Graph Generation
  5. 2022Label Semantic Knowledge Distillation for Unbiased Scene Graph Generation
  6. 2026Unveiling the Unknown: Open Vocabulary Object Detection with Scene Graphs
  7. 2025PRISM-0: A Predicate-Rich Scene Graph Generation Framework for Zero-Shot Open-Vocabulary Tasks
  8. 2024RA-SGG: Retrieval-Augmented Scene Graph Generation Framework via Multi-Prototype Learning