mlsanityData to 7 Oct 2026

Exploiting Scene Graphs for Human-Object Interaction Detection

arXiv 2108.08584 ↗First posted 19 Aug 2021cs.CVcited by 22

Not on the map. Why?

Abstract (arXiv)

Human-Object Interaction (HOI) detection is a fundamental visual task aiming at localizing and recognizing interactions between humans and objects. Existing works focus on the visual and linguistic features of humans and objects. However, they do not capitalise on the high-level and semantic relationships present in the image, which provides crucial contextual and detailed relational knowledge for HOI inference. We propose a novel method to exploit this information, through the scene graph, for the Human-Object Interaction (SG2HOI) detection task. Our method, SG2HOI, incorporates the SG information in two ways: (1) we embed a scene graph into a global context clue, serving as the scene-specific environmental context; and (2) we build a relation-aware message-passing module to gather relationships from objects' neighborhood and transfer them into interactions. Empirical evaluation shows that our SG2HOI method outperforms the state-of-the-art methods on two benchmark HOI datasets: V-COCO and HICO-DET. Code will be available at https://github.com/ht014/SG2HOI.

Key facts. Cited by 22 papers in the map (data through 7 Oct 2026). Builds on 23 of its 34 references that are in the map. First posted on arXiv 19 Aug 2021.

Builds on

23 of 34 references.

References · 23

  1. 2017Attention Is All You Needcited by 59,932
  2. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  3. 2014Microsoft COCO: Common Objects in Contextcited by 17,498
  4. 2015Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networkscited by 9,617
  5. 2016Visual Genome: Connecting Language and Vision Using Crowdsourced Dense Image Annotationscited by 2,646
  6. 2016ConceptNet 5.5: An Open Multilingual Graph of General Knowledgecited by 1,065
  7. 2017Scene Graph Generation by Iterative Message Passingcited by 462
  8. 2017Neural Motifs: Scene Graph Parsing with Global Contextcited by 413
  9. 2020Unbiased Scene Graph Generation from Biased Trainingcited by 355
  10. 2016Visual Relationship Detection with Language Priorscited by 333
  11. 2017Learning to Detect Human-Object Interactionscited by 238
  12. 2017Detecting and Recognizing Human-Object Interactionscited by 199
11 more
  1. 2015Visual Semantic Role Labelingcited by 188
  2. 2018Learning Human-Object Interactions by Graph Parsing Neural Networkscited by 176
  3. 2019PPDM: Parallel Point Detection and Matching for Real-time Human-Object Interaction Detectioncited by 143
  4. 2018iCAN: Instance-Centric Attention Network for Human-Object Interaction Detectioncited by 126
  5. 2020VSGNet: Spatial Attention Network for Detecting Human Object Interactions Using Graph Convolutionscited by 115
  6. 2017Visual Translation Embedding Network for Visual Relation Detectioncited by 112
  7. 2020Learning Human-Object Interaction Detection using Interaction Pointscited by 108
  8. 2017Detecting Visual Relationships with Deep Relational Networkscited by 97
  9. 2017Pixels to Graphs by Associative Embeddingcited by 58
  10. 2020Contextual Heterogeneous Graph Network for Human-Object Interaction Detectioncited by 50
  11. 2020Learning from the Scene and Borrowing from the Rich: Tackling the Long Tail in Scene Graph Generationcited by 22

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 6; 2023: 4; 2024: 2; 2025: 4; 2026: 6.

22 papers.

Sort:

By area:

  1. 2022GEN-VLKT: Simplify Association and Enhance Interaction Understanding for HOI Detectioncited by 119
  2. 2022Towards Open-vocabulary Scene Graph Generation with Prompt-based Finetuningcited by 54
  3. 2023Action Scene Graphs for Long-Form Understanding of Egocentric Videoscited by 51
  4. 2022Exploring Structure-aware Transformer over Interaction Proposals for Human-Object Interaction Detectioncited by 46
  5. 2022What to look at and where: Semantic and Spatial Refined Transformer for detecting human-object interactionscited by 35
  6. 2023Towards a Unified Transformer-based Framework for Scene Graph Generation and Human-object Interaction Detectioncited by 15
  7. 2022Chairs Can be Stood on: Overcoming Object Bias in Human-Object Interaction Detectioncited by 9
  8. 2024Towards Lifelong Scene Graph Generation with Knowledge-ware In-context Prompt Learningcited by 6
  9. 2024A Review of Human-Object Interaction Detectioncited by 5
  10. 2025Learning Human-Object Interaction as Groupscited by 5
  11. 20264DPC2hat: Towards Dynamic Point Cloud Understanding with Failure-Aware Bootstrappingcited by 5
  12. 2022A Skeleton-aware Graph Convolutional Network for Human-Object Interaction Detectioncited by 3
10 more
  1. 2025QueryCraft: Transformer-Guided Query Initialization for Enhanced Human-Object Interaction Detectioncited by 1
  2. 2025DQEN: Dual Query Enhancement Network for DETR-based HOI Detectioncited by 1
  3. 2023Mining Conditional Part Semantics with Occluded Extrapolation for Human-Object Interaction Detectioncited by 0
  4. 2023Head-Tail Cooperative Learning Network for Unbiased Scene Graph Generationcited by 0
  5. 2025Contextualized Representation Learning for Effective Human-Object Interaction Detectioncited by 0
  6. 2026CSA-Graphs: A Privacy-Preserving Structural Dataset for Child Sexual Abuse Researchcited by 0
  7. 2026CAGE-SGG: Counterfactual Active Graph Evidence for Open-Vocabulary Scene Graph Generationcited by 0
  8. 2026ReLIC-SGG: Relation Lattice Completion for Open-Vocabulary Scene Graph Generationcited by 0
  9. 2026ScriptHOI: Learning Scripted State Transitions for Open-Vocabulary Human-Object Interaction Detectioncited by 0
  10. 2026RelWitness: Open-Vocabulary 3D Scene Graph Generation with Visual-Geometric Relation Witnessescited by 0