mlsanityData to 7 Oct 2026

Enhancing Scene Graph Generation with Hierarchical Relationships and Commonsense Knowledge

arXiv 2311.12889 ↗First posted 21 Nov 2023cs.CV cs.AI cs.LGcited by 7

Not on the map. Why?

Abstract (arXiv)

This work introduces an enhanced approach to generating scene graphs by incorporating both a relationship hierarchy and commonsense knowledge. Specifically, we begin by proposing a hierarchical relation head that exploits an informative hierarchical structure. It jointly predicts the relation super-category between object pairs in an image, along with detailed relations under each super-category. Following this, we implement a robust commonsense validation pipeline that harnesses foundation models to critique the results from the scene graph prediction system, removing nonsensical predicates even with a small language-only model. Extensive experiments on Visual Genome and OpenImage V6 datasets demonstrate that the proposed modules can be seamlessly integrated as plug-and-play enhancements to existing scene graph generation algorithms. The results show significant improvements with an extensive set of reasonable predictions beyond dataset annotations. Codes are available at https://github.com/bowen-upenn/scene_graph_commonsense.

Key facts. Cited by 7 papers in the map (data through 7 Oct 2026). Builds on 69 of its 95 references that are in the map. First posted on arXiv 21 Nov 2023.

Builds on

69 of 95 references.

References · 69

  1. 2017Attention Is All You Needcited by 59,932
  2. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  3. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  4. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  5. 2020Language Models are Few-Shot Learnerscited by 32,199
  6. 2023GPT-4 Technical Reportcited by 18,132
  7. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  8. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,302
  9. 2016Semi-Supervised Classification with Graph Convolutional Networkscited by 11,282
  10. 2015Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networkscited by 9,617
  11. 2023Visual Instruction Tuningcited by 7,792
  12. 2020End-to-End Object Detection with Transformerscited by 7,384
38 more
  1. 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,069
  2. 2015You Only Look Once: Unified, Real-Time Object Detectioncited by 4,829
  3. 2023Improved Baselines with Visual Instruction Tuningcited by 4,362
  4. 2023Mistral 7Bcited by 3,065
  5. 2020Supervised Contrastive Learningcited by 2,803
  6. 2016Visual Genome: Connecting Language and Vision Using Crowdsourced Dense Image Annotationscited by 2,646
  7. 2023Sparks of Artificial General Intelligence: Early experiments with GPT-4cited by 2,579
  8. 2023A Survey on Hallucination in Large Language Models: Principles, Taxonomy, Challenges, and Open Questionscited by 1,582
  9. 2019Towards Robust Monocular Depth Estimation: Mixing Datasets for Zero-shot Cross-dataset Transfercited by 1,484
  10. 2014On the Properties of Neural Machine Translation: Encoder-Decoder Approachescited by 1,303
  11. 2023PaLM 2 Technical Reportcited by 1,154
  12. 2023CogVLM: Visual Expert for Pretrained Language Modelscited by 704
  13. 2018The Open Images Dataset V4: Unified image classification, object detection, and visual relationship detection at scalecited by 542
  14. 2017Scene Graph Generation by Iterative Message Passingcited by 462
  15. 2017Neural Motifs: Scene Graph Parsing with Global Contextcited by 413
  16. 2020Unbiased Scene Graph Generation from Biased Trainingcited by 355
  17. 2023Chameleon: Plug-and-Play Compositional Reasoning with Large Language Modelscited by 352
  18. 2016Visual Relationship Detection with Language Priorscited by 333
  19. 2018Graph R-CNN for Scene Graph Generationcited by 296
  20. 2018Learning to Compose Dynamic Tree Structures for Visual Contextscited by 258
  21. 2023Large Language Models as Commonsense Knowledge for Large-Scale Task Planningcited by 231
  22. 2020GPS-Net: Graph Property Sensing Network for Scene Graph Generationcited by 165
  23. 2019Knowledge-Embedded Routing Network for Scene Graph Generationcited by 163
  24. 2021Bipartite Graph Network with Adaptive Message Passing for Unbiased Scene Graph Generationcited by 137
  25. 2021A Comprehensive Survey of Scene Graphs: Generation and Applicationcited by 125
  26. 2017Visual Translation Embedding Network for Visual Relation Detectioncited by 112
  27. 2021Energy-Based Learning for Scene Graph Generationcited by 104
  28. 2021SGTR: End-to-end Scene Graph Generation with Transformercited by 102
  29. 2019Graphical Contrastive Losses for Scene Graph Parsingcited by 100
  30. 2022RelTR: Relation Transformer for Scene Graph Generationcited by 95
  31. 2022Scene Graph Generation: A Comprehensive Surveycited by 73
  32. 2021Learning to Generate Scene Graph from Natural Language Supervisioncited by 64
  33. 2020CogTree: Cognition Tree Loss for Unbiased Scene Graph Generationcited by 62
  34. 2022The Devil is in the Labels: Noisy Label Correction for Robust Scene Graph Generationcited by 60
  35. 2017Pixels to Graphs by Associative Embeddingcited by 58
  36. 2022Fine-Grained Scene Graph Generation with Data Transfercited by 57
  37. 2022Towards Open-vocabulary Scene Graph Generation with Prompt-based Finetuningcited by 54
  38. 2021Structured Sparse R-CNN for Direct Scene Graph Generationcited by 44

50 most cited of 69 All 69 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 1; 2025: 5; 2026: 1.

7 papers.

  1. 2024Evaluating Modern Approaches in 3D Scene Reconstruction: NeRF vs Gaussian-Based Methodscited by 16
  2. 2025Universal Scene Graph Generationcited by 11
  3. 2025Open World Scene Graph Generation using Vision Language Modelscited by 7
  4. 2025Dynamic Graph Induced Contour-aware Heat Conduction Network for Event-based Object Detectioncited by 1
  5. 2025Language-Driven Object-Oriented Two-Stage Method for Scene Graph Anticipationcited by 1
  6. 2025MSGFusion: Multimodal Scene Graph-Guided Infrared and Visible Image Fusioncited by 0
  7. 2026Visual Commonsense Driven Knowledge Refinements for Scene Graph Generationcited by 0