mlsanityData to 7 Oct 2026

From Pixels to Graphs: Open-Vocabulary Scene Graph Generation with Vision-Language Models

arXiv 2404.00906 ↗First posted 1 Apr 2024cs.CVcited by 43

On the map: Multimodal LLMs / vision-language models → VLM benchmarks and visual reasoning

Abstract (arXiv)

Scene graph generation (SGG) aims to parse a visual scene into an intermediate graph representation for downstream reasoning tasks. Despite recent advancements, existing methods struggle to generate scene graphs with novel visual relation concepts. To address this challenge, we introduce a new open-vocabulary SGG framework based on sequence generation. Our framework leverages vision-language pre-trained models (VLM) by incorporating an image-to-graph generation paradigm. Specifically, we generate scene graph sequences via image-to-text generation with VLM and then construct scene graphs from these sequences. By doing so, we harness the strong capabilities of VLM for open-vocabulary SGG and seamlessly integrate explicit relational modeling for enhancing the VL tasks. Experimental results demonstrate that our design not only achieves superior performance with an open vocabulary but also enhances downstream vision-language task performance through explicit relation modeling knowledge.

Key facts. Cited by 43 papers in the map (data through 7 Oct 2026). Placed in VLM benchmarks and visual reasoning, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 45 of its 56 references that are in the map. First posted on arXiv 1 Apr 2024.

Builds on

45 of 56 references.

Closest ancestors · 3

Same direction, newest first

  1. 2022PEVL: Position-enhanced Pre-training and Prompt Tuning for Vision-language Models23 May
  2. 2022BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generation28 Jan
  3. 2021UniTAB: Unifying Text and Box Outputs for Grounded Vision-Language Modeling23 Nov

Other references · 42

  1. 2019The Curious Case of Neural Text Degenerationcited by 2,881
  2. 2016Visual Genome: Connecting Language and Vision Using Crowdsourced Dense Image Annotationscited by 2,646
  3. 2015Microsoft COCO Captions: Data Collection and Evaluation Servercited by 1,447
  4. 2016Modeling Context in Referring Expressionscited by 1,054
  5. 2015Generation and Comprehension of Unambiguous Object Descriptionscited by 897
  6. 2016SPICE: Semantic Propositional Image Caption Evaluationcited by 820
  7. 2022OFA: Unifying Architectures, Tasks, and Modalities Through a Simple Sequence-to-Sequence Learning Frameworkcited by 742
  8. 2017Scene Graph Generation by Iterative Message Passingcited by 462
  9. 2017Neural Motifs: Scene Graph Parsing with Global Contextcited by 413
  10. 2020Unbiased Scene Graph Generation from Biased Trainingcited by 355
  11. 2018Learning to Compose Dynamic Tree Structures for Visual Contextscited by 258
  12. 2018Auto-Encoding Scene Graphs for Image Captioningcited by 254
30 more
  1. 2019Knowledge-Embedded Routing Network for Scene Graph Generationcited by 163
  2. 2021Bipartite Graph Network with Adaptive Message Passing for Unbiased Scene Graph Generationcited by 137
  3. 2016Graph-Structured Representations for Visual Question Answeringcited by 123
  4. 2022GEN-VLKT: Simplify Association and Enhance Interaction Understanding for HOI Detectioncited by 119
  5. 2022Panoptic Scene Graph Generationcited by 114
  6. 2019Learning by Abstraction: The Neural State Machinecited by 113
  7. 2018Explainable and Explicit Visual Reasoning over Scene Graphscited by 103
  8. 2021SGTR: End-to-end Scene Graph Generation with Transformercited by 102
  9. 2022RelTR: Relation Transformer for Scene Graph Generationcited by 95
  10. 2020Bridging Knowledge Graphs to Generate Scene Graphscited by 86
  11. 2021Learning to Generate Scene Graph from Natural Language Supervisioncited by 64
  12. 2022The Devil is in the Labels: Noisy Label Correction for Robust Scene Graph Generationcited by 60
  13. 2023HOICLIP: Efficient Knowledge Transfer for HOI Detection with Vision-Language Modelscited by 60
  14. 2022Fine-Grained Scene Graph Generation with Data Transfercited by 57
  15. 2022Towards Open-vocabulary Scene Graph Generation with Prompt-based Finetuningcited by 54
  16. 2021From General to Specific: Informative Scene Graph Generation via Balance Adjustmentcited by 51
  17. 2020Scene Graph Reasoning for Visual Question Answeringcited by 49
  18. 2021Structured Sparse R-CNN for Direct Scene Graph Generationcited by 44
  19. 2023Prototype-based Embedding Network for Scene Graph Generationcited by 37
  20. 2023Visually-Prompted Language Model for Fine-Grained Scene Graph Generation in an Open Worldcited by 35
  21. 2019Scene Graph Prediction with Limited Labelscited by 32
  22. 2021In Defense of Scene Graphs for Image Captioningcited by 31
  23. 2022RU-Net: Regularized Unrolling Network for Scene Graph Generationcited by 28
  24. 2021Visual Distant Supervision for Scene Graph Generationcited by 25
  25. 2022Iterative Scene Graph Generationcited by 24
  26. 2021ReFormer: The Relational Transformer for Image Captioningcited by 23
  27. 2018Scene Graph Parsing as Dependency Parsingcited by 21
  28. 2021Zero-Shot Scene Graph Relation Prediction through Commonsense Knowledge Integrationcited by 21
  29. 2022NICEST: Noisy Label Correction and Training for Robust Scene Graph Generationcited by 19
  30. 2021SGEITL: Scene Graph Enhanced Image-Text Learning for Visual Commonsense Reasoningcited by 16

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 4; 2025: 18; 2026: 21.

43 papers.

Sort:

By area:

  1. 2024HyperGLM: HyperGraph for Video Scene Graph Generation and Anticipationcited by 17
  2. 2026Graph is a Substrate Across Data Modalitiescited by 13
  3. 2025Compile Scene Graphs with Reinforcement Learningcited by 11
  4. 2025Open World Scene Graph Generation using Vision Language Modelscited by 7
  5. 2025Open-Vocabulary HOI Detection with Interaction-aware Prompt and Concept Calibrationcited by 7
  6. 2024ROOT: VLM based System for Indoor Scene Understanding and Beyondcited by 6
  7. 2025Learning 4D Panoptic Scene Graph Generation from Rich 2D Visual Scenecited by 6
  8. 2025Synthetic Visual Genomecited by 6
  9. 2025PRISM-0: A Predicate-Rich Scene Graph Generation Framework for Zero-Shot Open-Vocabulary Taskscited by 5
  10. 2025Agent Identity Evals: Measuring Agentic Identitycited by 5
  11. 2026GraphThinker: Reinforcing Temporally Grounded Video Reasoning with Event Graph Thinkingcited by 5
  12. 2026The Yes-Man Syndrome: Benchmarking Abstention in Embodied Robotic Agentscited by 4
31 more
  1. 2024BCTR: Bidirectional Conditioning Transformer for Scene Graph Generationcited by 3
  2. 2024End-to-end Open-vocabulary Video Visual Relationship Detection using Multi-modal Promptingcited by 3
  3. 2025From Objects to Events: Unlocking Complex Visual Understanding in Object Detectors via LLM-guided Symbolic Reasoningcited by 3
  4. 2025Robo-SGG: Exploiting Layout-Oriented Normalization and Restitution Can Improve Robust Scene Graph Generationcited by 3
  5. 2025Harnessing Caption Detailness for Data-Efficient Text-to-Image Generationcited by 3
  6. 2025RBT4DNN: Requirements-based Testing of Neural Networkscited by 2
  7. 2025From Data to Modeling: Fully Open-vocabulary Scene Graph Generationcited by 2
  8. 2025CORE-3D: Context-aware Open-vocabulary Retrieval by Embeddings in 3Dcited by 2
  9. 2025Attention over Scene Graphs: Indoor Scene Representations Toward CSAI Classificationcited by 2
  10. 2026360deg Image Perception with MLLMs: A Comprehensive Benchmark and a Training-Free Methodcited by 2
  11. 2026Streamlined Open-Vocabulary Human-Object Interaction Detectioncited by 2
  12. 2026IMPACT-CYCLE: A Contract-Based Multi-Agent System for Claim-Level Supervisory Correction of Long-Video Semantic Memorycited by 2
  13. 2025Language-Guided Object Search in Agricultural Environmentscited by 1
  14. 2026SceneGraphVLM: Dynamic Scene Graph Generation from Video with Vision-Language Modelscited by 1
  15. 2026HyperVis: Continuous Latent Visual Relational Graphs on the Lorentz Hyperboloid for Compositional Reasoningcited by 1
  16. 2026Graph it first! Enabling Reasoning on Long-form Egocentric Videos through Scene Graphscited by 1
  17. 2025Generalized Visual Relation Detection with Diffusion Modelscited by 0
  18. 2025Interaction-Centric Knowledge Infusion and Transfer for Open-Vocabulary Scene Graph Generationcited by 0
  19. 2025Generative Human-Object Interaction Detection via Differentiable Cognitive Steering of Multi-modal LLMscited by 0
  20. 2026PRISM-CAFO: Prior-conditioned Remote-sensing Infrastructure Segmentation and Mapping for CAFOscited by 0
  21. 2026BUSSARD: Normalizing Flows for Bijective Universal Scene-Specific Anomalous Relationship Detectioncited by 0
  22. 2026Training-Only Heterogeneous Image-Patch-Text Graph Supervision for Advancing Few-Shot Learning Adapterscited by 0
  23. 2026Cross-Domain Demo-to-Code via Neurosymbolic Counterfactual Reasoningcited by 0
  24. 2026CSA-Graphs: A Privacy-Preserving Structural Dataset for Child Sexual Abuse Researchcited by 0
  25. 2026HSG: Hyperbolic Scene Graphcited by 0
  26. 2026Can We Build Scene Graphs, Not Classify Them? FlowSG: Progressive Image-Conditioned Scene Graph Generation with Flow Matchingcited by 0
  27. 2026CAGE-SGG: Counterfactual Active Graph Evidence for Open-Vocabulary Scene Graph Generationcited by 0
  28. 2026ReLIC-SGG: Relation Lattice Completion for Open-Vocabulary Scene Graph Generationcited by 0
  29. 2026Modeling Scientific Experiment Scenes: Dataset and Modelcited by 0
  30. 2026OvDSGG: End-to-End Open-Vocabulary Dynamic Scene Graph Generationcited by 0
  31. 2026PROBE: Manipulation-Grounded Visual Question Answering with VLM Agentscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Measuring Image-Relation Alignment: Reference-Free Evaluation of VLMs and Synthetic Pre-training for Open-Vocabulary Scene Graph Generation
  2. 2024LLaVA-SpaceSGG: Visual Instruct Tuning for Open-vocabulary Scene Graph Generation with Enhanced Spatial Relations
  3. 2026SGG-Rrm 3: From Next-Token Prediction to End-to-End Unbiased Scene Graph Generation
  4. 2025VOST-SGG: VLM-Aided One-Stage Spatio-Temporal Scene Graph Generation
  5. 2024Scene Graph Generation with Role-Playing Large Language Models
  6. 2024Predicate Debiasing in Vision-Language Models Integration for Scene Graph Generation Enhancement
  7. 2024Generate Any Scene: Scene Graph Driven Data Synthesis for Visual Generation Training
  8. 2024Relation-aware Hierarchical Prompt for Open-vocabulary Scene Graph Generation