# Contextual Heterogeneous Graph Network for Human-Object Interaction Detection

arXiv: https://arxiv.org/abs/2010.10001 · HTML: https://mlsanity.com/paper/2010.10001/

## Key facts

- Cited by 50 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 20 Oct 2020.

## Built on by (most cited first, 30 of 50)

- [HOTR: End-to-End Human-Object Interaction Detection with Transformers](https://mlsanity.com/paper/2104.13682/) (2021)
- [QPIC: Query-Based Pairwise Human-Object Interaction Detection with Image-Wide Contextual Information](https://mlsanity.com/paper/2103.05399/) (2021)
- [End-to-End Human Object Interaction Detection with HOI Transformer](https://mlsanity.com/paper/2103.04503/) (2021)
- [GEN-VLKT: Simplify Association and Enhance Interaction Understanding for HOI Detection](https://mlsanity.com/paper/2203.13954/) (2022)
- [Mining the Benefits of Two-stage and One-stage HOI Detection](https://mlsanity.com/paper/2108.05077/) (2021)
- [Exploring Predicate Visual Context in Detecting Human-Object Interactions](https://mlsanity.com/paper/2308.06202/) (2023)
- [Multimodal learning with graphs](https://mlsanity.com/paper/2209.03299/) (2022)
- [Exploring Structure-aware Transformer over Interaction Proposals for Human-Object Interaction Detection](https://mlsanity.com/paper/2206.06291/) (2022)
- [ViPLO: Vision Transformer based Pose-Conditioned Self-Loop Graph for Human-Object Interaction Detection](https://mlsanity.com/paper/2304.08114/) (2023)
- [Interactiveness Field in Human-Object Interactions](https://mlsanity.com/paper/2204.07718/) (2022)
- [Human-Object Interaction Detection via Disentangled Transformer](https://mlsanity.com/paper/2204.09290/) (2022)
- [Relational Context Learning for Human-Object Interaction Detection](https://mlsanity.com/paper/2304.04997/) (2023)
- [What to look at and where: Semantic and Spatial Refined Transformer for detecting human-object interactions](https://mlsanity.com/paper/2204.00746/) (2022)
- [MSTR: Multi-Scale Transformer for End-to-End Human-Object Interaction Detection](https://mlsanity.com/paper/2203.14709/) (2022)
- [Visual Relationship Detection Using Part-and-Sum Transformers with Composite Queries](https://mlsanity.com/paper/2105.02170/) (2021)
- [Open-World Human-Object Interaction Detection via Multi-modal Prompts](https://mlsanity.com/paper/2406.07221/) (2024)
- [Neural-Logic Human-Object Interaction Detection](https://mlsanity.com/paper/2311.09817/) (2023)
- [Exploiting Scene Graphs for Human-Object Interaction Detection](https://mlsanity.com/paper/2108.08584/) (2021)
- [Detecting Human-Object Interactions with Object-Guided Cross-Modal Calibrated Semantics](https://mlsanity.com/paper/2202.00259/) (2022)
- [Boosting Human-Object Interaction Detection with Text-to-Image Diffusion Model](https://mlsanity.com/paper/2305.12252/) (2023)
- [Category Query Learning for Human-Object Interaction Classification](https://mlsanity.com/paper/2303.14005/) (2023)
- [Iwin: Human-Object Interaction Detection via Transformer with Irregular Windows](https://mlsanity.com/paper/2203.10537/) (2022)
- [Geometric Features Informed Multi-person Human-object Interaction Recognition in Videos](https://mlsanity.com/paper/2207.09425/) (2022)
- [Consistency Learning via Decoding Path Augmentation for Transformers in Human Object Interaction Detection](https://mlsanity.com/paper/2204.04836/) (2022)
- [Improving Human-Object Interaction Detection via Phrase Learning and Label Composition](https://mlsanity.com/paper/2112.07383/) (2021)
- [Towards a Unified Transformer-based Framework for Scene Graph Generation and Human-object Interaction Detection](https://mlsanity.com/paper/2311.01755/) (2023)
- [Human-Object Interaction Detection Collaborated with Large Relation-driven Diffusion Models](https://mlsanity.com/paper/2410.20155/) (2024)
- [Nonverbal Interaction Detection](https://mlsanity.com/paper/2407.08133/) (2024)
- [Video-based Human-Object Interaction Detection from Tubelet Tokens](https://mlsanity.com/paper/2206.01908/) (2022)
- [Unified Visual Relationship Detection with Vision and Language Models](https://mlsanity.com/paper/2303.08998/) (2023)

## Abstract (arXiv)

Human-object interaction(HOI) detection is an important task for understanding human activity. Graph structure is appropriate to denote the HOIs in the scene. Since there is an subordination between human and object---human play subjective role and object play objective role in HOI, the relations between homogeneous entities and heterogeneous entities in the scene should also not be equally the same. However, previous graph models regard human and object as the same kind of nodes and do not consider that the messages are not equally the same between different entities. In this work, we address such a problem for HOI task by proposing a heterogeneous graph network that models humans and objects as different kinds of nodes and incorporates intra-class messages between homogeneous nodes and inter-class messages between heterogeneous nodes. In addition, a graph attention mechanism based on the intra-class context and inter-class context is exploited to improve the learning. Extensive experiments on the benchmark datasets V-COCO and HICO-DET demonstrate that the intra-class and inter-class messages are very important in HOI detection and verify the effectiveness of our method.