mlsanityData to 7 Oct 2026

Zero-Shot Scene Graph Relation Prediction through Commonsense Knowledge Integration

arXiv 2107.05080 ↗First posted 11 Jul 2021cs.CV cs.LGcited by 21

Not on the map. Why?

Abstract (arXiv)

Relation prediction among entities in images is an important step in scene graph generation (SGG), which further impacts various visual understanding and reasoning tasks. Existing SGG frameworks, however, require heavy training yet are incapable of modeling unseen (i.e.,zero-shot) triplets. In this work, we stress that such incapability is due to the lack of commonsense reasoning,i.e., the ability to associate similar entities and infer similar relations based on general understanding of the world. To fill this gap, we propose CommOnsense-integrAted sCenegrapHrElation pRediction (COACHER), a framework to integrate commonsense knowledge for SGG, especially for zero-shot relation prediction. Specifically, we develop novel graph mining pipelines to model the neighborhoods and paths around entities in an external commonsense knowledge graph, and integrate them on top of state-of-the-art SGG frameworks. Extensive quantitative evaluations and qualitative case studies on both original and manipulated datasets from Visual Genome demonstrate the effectiveness of our proposed approach.

Key facts. Cited by 21 papers in the map (data through 7 Oct 2026). Builds on 27 of its 36 references that are in the map. First posted on arXiv 11 Jul 2021.

Builds on

27 of 36 references.

References · 27

  1. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  2. 2015Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networkscited by 9,617
  3. 2017Neural Message Passing for Quantum Chemistrycited by 3,460
  4. 2015SSD: Single Shot MultiBox Detectorcited by 3,125
  5. 2015Fast R-CNNcited by 2,930
  6. 2018YOLOv3: An Incremental Improvementcited by 2,652
  7. 2016Visual Genome: Connecting Language and Vision Using Crowdsourced Dense Image Annotationscited by 2,646
  8. 2016ConceptNet 5.5: An Open Multilingual Graph of General Knowledgecited by 1,065
  9. 2017Scene Graph Generation by Iterative Message Passingcited by 462
  10. 2017Neural Motifs: Scene Graph Parsing with Global Contextcited by 413
  11. 2020Unbiased Scene Graph Generation from Biased Trainingcited by 355
  12. 2016Visual Relationship Detection with Language Priorscited by 333
15 more
  1. 2018Auto-Encoding Scene Graphs for Image Captioningcited by 254
  2. 2018Exploring Visual Relationship for Image Captioningcited by 215
  3. 2019Controllable Text-to-Image Generationcited by 187
  4. 2019ManiGAN: Text-Guided Image Manipulationcited by 156
  5. 2016Graph-Structured Representations for Visual Question Answeringcited by 123
  6. 2019Scene Graph Generation with External Knowledge and Image Reconstructioncited by 102
  7. 2016Time for a change: a tutorial for comparing multiple classifiers through Bayesian analysiscited by 92
  8. 2013Loopy Belief Propagation for Approximate Inference: An Empirical Studycited by 76
  9. 2017Visual Relationship Detection with Internal and External Linguistic Knowledge Distillationcited by 60
  10. 2017Pixels to Graphs by Associative Embeddingcited by 58
  11. 2016Image Captioning and Visual Question Answering Based on Attributes and External Knowledgecited by 44
  12. 2020Lightweight Generative Adversarial Networks for Text-Guided Image Manipulationcited by 33
  13. 2017Stack-Captioning: Coarse-to-Fine Learning for Image Captioningcited by 27
  14. 2017Incorporating External Knowledge to Answer Open-Domain Visual Questions with Dynamic Memory Networkscited by 12
  15. 2016On Support Relations and Semantic Scene Graphscited by 9

Built on by

Cited by year

Citing papers per year: 2023: 18; 2024: 2; 2025: 1.

21 papers.

Sort:

By area:

  1. 2024From Pixels to Graphs: Open-Vocabulary Scene Graph Generation with Vision-Language Modelscited by 43
  2. 2023Visually-Prompted Language Model for Fine-Grained Scene Graph Generation in an Open Worldcited by 35
  3. 2023AutoReP: Automatic ReLU Replacement for Fast Private Network Inferencecited by 29
  4. 2023Zero-shot Visual Relation Detection via Composite Visual Cues from Large Language Modelscited by 27
  5. 2023ReGen: Zero-Shot Text Classification via Training Data Generation with Progressive Dense Retrievalcited by 26
  6. 2023LinGCN: Structural Linearized Graph Convolutional Network for Homomorphically Encrypted Inferencecited by 26
  7. 2023Compositional Prompt Tuning with Motion Cues for Open-vocabulary Video Relation Detectioncited by 22
  8. 2023Accel-GCN: High-Performance GPU Accelerator Design for Graph Convolution Networkscited by 22
  9. 2023Knowledge-Infused Prompting: Assessing and Advancing Clinical Text Data Generation with Large Language Modelscited by 20
  10. 2024OpenPSG: Open-set Panoptic Scene Graph Generation via Large Multimodal Modelscited by 15
  11. 2023Neighborhood-Regularized Self-Training for Learning with Few Labelscited by 14
  12. 2023A Review on Knowledge Graphs for Healthcare: Resources, Applications, and Promisescited by 13
9 more
  1. 2023RRNet: Towards ReLU-Reduced Neural Network for Two-party Computation Based Private Inferencecited by 12
  2. 2023R-Mixup: Riemannian Mixup for Biological Networkscited by 8
  3. 2023Open Visual Knowledge Extraction via Relation-Oriented Multimodality Model Promptingcited by 5
  4. 2023You Only Transfer What You Share: Intersection-Induced Graph Transfer Learning for Link Predictioncited by 4
  5. 2023Fine-Grained is Too Coarse: A Novel Data-Centric Approach for Efficient Scene Graph Generationcited by 4
  6. 2023PV2TEA: Patching Visual Modality to Textual-Established Information Extractioncited by 2
  7. 2023Informative Scene Graph Generation via Debiasingcited by 2
  8. 2023HICO-DET-SG and V-COCO-SG: New Data Splits for Evaluating the Systematic Generalization Performance of Human-Object Interaction Detection Modelscited by 1
  9. 2025Augmented Vision-Language Models: A Systematic Reviewcited by 0