mlsanityData to 14 Oct 2026

Towards Open-vocabulary Scene Graph Generation with Prompt-based Finetuning

arXiv 2208.08165 ↗First posted 17 Aug 2022cs.CVcited by 54

Not on the map. Why?

Abstract (arXiv)

Scene graph generation (SGG) is a fundamental task aimed at detecting visual relations between objects in an image. The prevailing SGG methods require all object classes to be given in the training set. Such a closed setting limits the practical application of SGG. In this paper, we introduce open-vocabulary scene graph generation, a novel, realistic and challenging setting in which a model is trained on a set of base object classes but is required to infer relations for unseen target object classes. To this end, we propose a two-step method that firstly pre-trains on large amounts of coarse-grained region-caption data and then leverages two prompt-based techniques to finetune the pre-trained model without updating its parameters. Moreover, our method can support inference over completely unseen object classes, which existing methods are incapable of handling. On extensive experiments on three benchmark datasets, Visual Genome, GQA, and Open-Image, our method significantly outperforms recent, strong SGG methods on the setting of Ov-SGG, as well as on the conventional closed SGG.

Key facts. Cited by 54 papers in the map (data through 14 Oct 2026). Builds on 53 of its 62 references that are in the map. First posted on arXiv 17 Aug 2022.

Builds on

53 of 62 references.

References · 53

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  2. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  3. 2020Language Models are Few-Shot Learnerscited by 32,419
  4. 2014Microsoft COCO: Common Objects in Contextcited by 17,608
  5. 2016Semi-Supervised Classification with Graph Convolutional Networkscited by 11,361
  6. 2015Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networkscited by 9,639
  7. 2021Prefix-Tuning: Optimizing Continuous Prompts for Generationcited by 3,915
  8. 2016Visual Genome: Connecting Language and Vision Using Crowdsourced Dense Image Annotationscited by 2,656
  9. 2015VQA: Visual Question Answeringcited by 2,506
  10. 2018Deep contextualized word representationscited by 2,290
  11. 2021Pre-train, Prompt, and Predict: A Systematic Survey of Prompting Methods in Natural Language Processingcited by 2,234
  12. 2019Language Models as Knowledge Bases?cited by 1,936
38 more
  1. 2020Oscar: Object-Semantics Aligned Pre-training for Vision-Language Taskscited by 1,214
  2. 2019VL-BERT: Pre-training of Generic Visual-Linguistic Representationscited by 954
  3. 2020Contrastive Learning of Medical Visual Representations from Paired Images and Textcited by 641
  4. 2018From Recognition to Cognition: Visual Commonsense Reasoningcited by 556
  5. 2018The Open Images Dataset V4: Unified image classification, object detection, and visual relationship detection at scalecited by 544
  6. 2017Scene Graph Generation by Iterative Message Passingcited by 462
  7. 2020Open-Vocabulary Object Detection Using Captionscited by 427
  8. 2017Neural Motifs: Scene Graph Parsing with Global Contextcited by 414
  9. 2021Learning How to Ask: Querying LMs with Mixtures of Soft Promptscited by 377
  10. 2020Unbiased Scene Graph Generation from Biased Trainingcited by 356
  11. 2016Visual Relationship Detection with Language Priorscited by 334
  12. 2020Pixel-BERT: Aligning Image Pixels with Text by Deep Multi-Modal Transformerscited by 288
  13. 20193D Scene Graph: A Structure for Unified Semantics, 3D Space, and Cameracited by 281
  14. 2021TransVG: End-to-End Visual Grounding with Transformerscited by 271
  15. 2018Learning to Compose Dynamic Tree Structures for Visual Contextscited by 259
  16. 2018Auto-Encoding Scene Graphs for Image Captioningcited by 255
  17. 2018A Simple Method for Commonsense Reasoningcited by 223
  18. 2018Exploring Visual Relationship for Image Captioningcited by 215
  19. 2019Commonsense Knowledge Mining from Pretrained Modelscited by 189
  20. 2020GPS-Net: Graph Property Sensing Network for Scene Graph Generationcited by 166
  21. 2019Knowledge-Embedded Routing Network for Scene Graph Generationcited by 164
  22. 2021Bipartite Graph Network with Adaptive Message Passing for Unbiased Scene Graph Generationcited by 137
  23. 2016Graph-Structured Representations for Visual Question Answeringcited by 123
  24. 2017Visual Translation Embedding Network for Visual Relation Detectioncited by 112
  25. 2021Energy-Based Learning for Scene Graph Generationcited by 104
  26. 2020Bridging Knowledge Graphs to Generate Scene Graphscited by 86
  27. 2021Learning to Generate Scene Graph from Natural Language Supervisioncited by 64
  28. 2020VD-BERT: A Unified Vision and Dialog Transformer with BERTcited by 55
  29. 2021Exploiting Edge-Oriented Reasoning for 3D Point-based Scene Graph Analysiscited by 49
  30. 2018Zero-Shot Object Detection: Learning to Simultaneously Recognize and Localize Novel Conceptscited by 48
  31. 2017Weakly-supervised learning of visual relationscited by 36
  32. 2020Graph Density-Aware Losses for Novel Compositions in Scene Graph Generationcited by 31
  33. 2021Linguistic Structures as Weak Supervision for Visual Scene Graph Generationcited by 29
  34. 2020Weakly Supervised Visual Semantic Parsingcited by 28
  35. 2020Learning from the Scene and Borrowing from the Rich: Tackling the Long Tail in Scene Graph Generationcited by 22
  36. 2021Exploiting Scene Graphs for Human-Object Interaction Detectioncited by 22
  37. 2020Tackling the Unannotated: Scene Graph Generation with Bias-Reduced Modelscited by 19
  38. 2021Image Scene Graph Generation (SGG) Benchmarkcited by 15

50 most cited of 53 All 53 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 15; 2024: 10; 2025: 13; 2026: 16.

54 papers.

Sort:

By area:

  1. 2023Read-only Prompt Optimization for Vision-Language Few-shot Learningcited by 58
  2. 2024From Pixels to Graphs: Open-Vocabulary Scene Graph Generation with Vision-Language Modelscited by 43
  3. 2023Visually-Prompted Language Model for Fine-Grained Scene Graph Generation in an Open Worldcited by 35
  4. 2023LLM4SGG: Large Language Models for Weakly Supervised Scene Graph Generationcited by 30
  5. 2023Expanding Scene Graph Boundaries: Fully Open-vocabulary Scene Graph Generation via Visual-Concept Alignment and Retentioncited by 29
  6. 2023Zero-shot Visual Relation Detection via Composite Visual Cues from Large Language Modelscited by 27
  7. 2023Compositional Prompt Tuning with Motion Cues for Open-vocabulary Video Relation Detectioncited by 22
  8. 2024HyperGLM: HyperGraph for Video Scene Graph Generation and Anticipationcited by 19
  9. 2023Probabilistic Prompt Learning for Dense Predictioncited by 17
  10. 2025Knowledge-Aligned Counterfactual-Enhancement Diffusion Perception for Unsupervised Cross-Domain Visual Emotion Recognitioncited by 17
  11. 2024OpenPSG: Open-set Panoptic Scene Graph Generation via Large Multimodal Modelscited by 16
  12. 2024Scene Graph Generation with Role-Playing Large Language Modelscited by 16
38 more
  1. 2023VLPrompt: Vision-Language Prompting for Panoptic Scene Graph Generationcited by 12
  2. 2025Safe-Sora: Safe Text-to-Video Generation via Graphical Watermarkingcited by 11
  3. 2023Open-Vocabulary Object Detection via Scene Graph Discoverycited by 7
  4. 2023Enhancing Scene Graph Generation with Hierarchical Relationships and Commonsense Knowledgecited by 7
  5. 2024Towards Lifelong Scene Graph Generation with Knowledge-ware In-context Prompt Learningcited by 6
  6. 2024Scene-Graph ViT: End-to-End Open-Vocabulary Visual Relationship Detectioncited by 6
  7. 2024Tri-modal Confluence with Temporal Dynamics for Scene Graph Generation in Operating Roomscited by 6
  8. 2023Skew Class-balanced Re-weighting for Unbiased Scene Graph Generationcited by 5
  9. 2023Open Visual Knowledge Extraction via Relation-Oriented Multimodality Model Promptingcited by 5
  10. 2025PRISM-0: A Predicate-Rich Scene Graph Generation Framework for Zero-Shot Open-Vocabulary Taskscited by 5
  11. 2026GraphThinker: Reinforcing Temporally Grounded Video Reasoning with Event Graph Thinkingcited by 5
  12. 2023Decomposed Prototype Learning for Few-Shot Scene Graph Generationcited by 4
  13. 2024End-to-end Open-vocabulary Video Visual Relationship Detection using Multi-modal Promptingcited by 3
  14. 2025From Data to Modeling: Fully Open-vocabulary Scene Graph Generationcited by 2
  15. 2025A Retrospect to Multi-prompt Learning across Vision and Languagecited by 2
  16. 2024Predicate Debiasing in Vision-Language Models Integration for Scene Graph Generation Enhancementcited by 1
  17. 2025X-SG2S: Safe and Generalizable Gaussian Splatting with X-dimensional Watermarkscited by 1
  18. 2025Measuring Image-Relation Alignment: Reference-Free Evaluation of VLMs and Synthetic Pre-training for Open-Vocabulary Scene Graph Generationcited by 1
  19. 2025ZING-3D: Zero-shot Incremental 3D Scene Graphs via Vision-Language Modelscited by 1
  20. 2025ClusterStyle: Modeling Intra-Style Diversity with Prototypical Clustering for Stylized Motion Generationcited by 1
  21. 2026SpatialMem: Metric-Aligned Long-Horizon Video Memory for Language Grounding and QAcited by 1
  22. 2026Revisiting Weakly-Supervised Video Scene Graph Generation via Pair Affinity Learningcited by 1
  23. 2026ScriptHOI: Learning Scripted State Transitions for Open-Vocabulary Human-Object Interaction Detectioncited by 1
  24. 2026HNC: Leveraging Hard Negative Captions towards Models with Fine-Grained Visual-Linguistic Comprehension Capabilitiescited by 1
  25. 2026PhysScene: A Scene Graph Dataset for Scientific Visual Reasoning in Physics Experimentscited by 1
  26. 2023Head-Tail Cooperative Learning Network for Unbiased Scene Graph Generationcited by 0
  27. 2023ALF: Adaptive Label Finetuning for Scene Graph Generationcited by 0
  28. 2024Enhanced Data Transfer Cooperating with Artificial Triplets for Scene Graph Generationcited by 0
  29. 2025Open-Vocabulary Semantic Segmentation with Uncertainty Alignment for Robotic Scene Understanding in Indoor Building Environmentscited by 0
  30. 2025Generalized Visual Relation Detection with Diffusion Modelscited by 0
  31. 2025ART: Adaptive Relation Tuning for Generalized Relation Predictioncited by 0
  32. 2025Interaction-Centric Knowledge Infusion and Transfer for Open-Vocabulary Scene Graph Generationcited by 0
  33. 2026DVAR: Adversarial Multi-Agent Debate for Video Authenticity Detectioncited by 0
  34. 2026Can We Build Scene Graphs, Not Classify Them? FlowSG: Progressive Image-Conditioned Scene Graph Generation with Flow Matchingcited by 0
  35. 2026CAGE-SGG: Counterfactual Active Graph Evidence for Open-Vocabulary Scene Graph Generationcited by 0
  36. 2026ReLIC-SGG: Relation Lattice Completion for Open-Vocabulary Scene Graph Generationcited by 0
  37. 2026RelWitness: Open-Vocabulary 3D Scene Graph Generation with Visual-Geometric Relation Witnessescited by 0
  38. 2026Resolving Ambiguity in Composed Image Retrieval via Calibrated Interactioncited by 0

50 most cited of 54 All 54 (JSON)