# Relational Context Learning for Human-Object Interaction Detection

arXiv: https://arxiv.org/abs/2304.04997 · HTML: https://mlsanity.com/paper/2304.04997/

## Key facts

- Cited by 36 papers in the map (data through 14 Oct 2026).
- Builds on 36 of its 49 references that are in the map.
- First posted on arXiv 11 Apr 2023.

## Builds on (most cited first, 30 of 36)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Microsoft COCO: Common Objects in Context](https://mlsanity.com/paper/1405.0312/) (2014)
- [Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks](https://mlsanity.com/paper/1506.01497/) (2015)
- [End-to-End Object Detection with Transformers](https://mlsanity.com/paper/2005.12872/) (2020)
- [Deformable DETR: Deformable Transformers for End-to-End Object Detection](https://mlsanity.com/paper/2010.04159/) (2020)
- [Generalized Intersection over Union: A Metric and A Loss for Bounding Box Regression](https://mlsanity.com/paper/1902.09630/) (2019)
- [Learning to Detect Human-Object Interactions](https://mlsanity.com/paper/1702.05448/) (2017)
- [Exploring Visual Relationship for Image Captioning](https://mlsanity.com/paper/1809.07041/) (2018)
- [Visual Semantic Role Labeling](https://mlsanity.com/paper/1505.04474/) (2015)
- [HOTR: End-to-End Human-Object Interaction Detection with Transformers](https://mlsanity.com/paper/2104.13682/) (2021)
- [Learning Human-Object Interactions by Graph Parsing Neural Networks](https://mlsanity.com/paper/1808.07962/) (2018)
- [Image Captioning: Transforming Objects into Words](https://mlsanity.com/paper/1906.05963/) (2019)
- [QPIC: Query-Based Pairwise Human-Object Interaction Detection with Image-Wide Contextual Information](https://mlsanity.com/paper/2103.05399/) (2021)
- [PPDM: Parallel Point Detection and Matching for Real-time Human-Object Interaction Detection](https://mlsanity.com/paper/1912.12898/) (2019)
- [End-to-End Human Object Interaction Detection with HOI Transformer](https://mlsanity.com/paper/2103.04503/) (2021)
- [Visual Compositional Learning for Human-Object Interaction Detection](https://mlsanity.com/paper/2007.12407/) (2020)
- [iCAN: Instance-Centric Attention Network for Human-Object Interaction Detection](https://mlsanity.com/paper/1808.10437/) (2018)
- [DRG: Dual Relation Graph for Human-Object Interaction Detection](https://mlsanity.com/paper/2008.11714/) (2020)
- [VSGNet: Spatial Attention Network for Detecting Human Object Interactions Using Graph Convolutions](https://mlsanity.com/paper/2003.05541/) (2020)
- [Mining the Benefits of Two-stage and One-stage HOI Detection](https://mlsanity.com/paper/2108.05077/) (2021)
- [Reformulating HOI Detection as Adaptive Set Prediction](https://mlsanity.com/paper/2103.05983/) (2021)
- [Spatially Conditioned Graphs for Detecting Human-Object Interactions](https://mlsanity.com/paper/2012.06060/) (2020)
- [Efficient Two-Stage Detection of Human-Object Interactions with a Novel Unary-Pairwise Transformer](https://mlsanity.com/paper/2112.01838/) (2021)
- [HOI Analysis: Integrating and Decomposing Human-Object Interaction](https://mlsanity.com/paper/2010.16219/) (2020)
- [Glance and Gaze: Inferring Action-aware Points for One-Stage Human-Object Interaction Detection](https://mlsanity.com/paper/2104.05269/) (2021)
- [Contextual Heterogeneous Graph Network for Human-Object Interaction Detection](https://mlsanity.com/paper/2010.10001/) (2020)
- [Exploring Structure-aware Transformer over Interaction Proposals for Human-Object Interaction Detection](https://mlsanity.com/paper/2206.06291/) (2022)
- [ConsNet: Learning Consistency Graph for Zero-Shot Human-Object Interaction Detection](https://mlsanity.com/paper/2008.06254/) (2020)
- [A Structured Model For Action Detection](https://mlsanity.com/paper/1812.03544/) (2018)
- [Human-Object Interaction Detection via Disentangled Transformer](https://mlsanity.com/paper/2204.09290/) (2022)

## Built on by (most cited first, 30 of 36)

- [Detecting Any Human-Object Interaction Relationship: Universal HOI Detector with Spatial Prompt Learning on Foundation Models](https://mlsanity.com/paper/2311.03799/) (2023)
- [Open-World Human-Object Interaction Detection via Multi-modal Prompts](https://mlsanity.com/paper/2406.07221/) (2024)
- [EZ-HOI: VLM Adaptation via Guided Prompt Learning for Zero-Shot HOI Detection](https://mlsanity.com/paper/2410.23904/) (2024)
- [Exploring Conditional Multi-Modal Prompts for Zero-shot HOI Detection](https://mlsanity.com/paper/2408.02484/) (2024)
- [Exploring the Potential of Large Foundation Models for Open-Vocabulary HOI Detection](https://mlsanity.com/paper/2404.06194/) (2024)
- [Exo2Ego: Exocentric Knowledge Guided MLLM for Egocentric Video Understanding](https://mlsanity.com/paper/2503.09143/) (2025)
- [Zero-shot Referring Expression Comprehension via Structural Similarity Between Images and Captions](https://mlsanity.com/paper/2311.17048/) (2023)
- [Learning from Observer Gaze:Zero-Shot Attention Prediction Oriented by Human-Object Interaction Recognition](https://mlsanity.com/paper/2405.09931/) (2024)
- [Locality-Aware Zero-Shot Human-Object Interaction Detection](https://mlsanity.com/paper/2505.19503/) (2025)
- [HOLa: Zero-Shot HOI Detection with Low-Rank Decomposed VLM Feature Adaptation](https://mlsanity.com/paper/2507.15542/) (2025)
- [Diagnosing Human-object Interaction Detectors](https://mlsanity.com/paper/2308.08529/) (2023)
- [Open-Vocabulary HOI Detection with Interaction-aware Prompt and Concept Calibration](https://mlsanity.com/paper/2508.03207/) (2025)
- [What if Agents Could Imagine? Reinforcing Open-Vocabulary HOI Comprehension through Generation](https://mlsanity.com/paper/2602.11499/) (2026)
- [A Review of Human-Object Interaction Detection](https://mlsanity.com/paper/2408.10641/) (2024)
- [CrossHOI-Bench: A Unified Benchmark for HOI Evaluation across Vision-Language Models and HOI-Specific Methods](https://mlsanity.com/paper/2508.18753/) (2025)
- [Focusing on what to decode and what to train: SOV Decoding with Specific Target Guided DeNoising and Vision Language Advisor](https://mlsanity.com/paper/2307.02291/) (2023)
- [Towards Zero-shot Human-Object Interaction Detection via Vision-Language Integration](https://mlsanity.com/paper/2403.07246/) (2024)
- [Enhancing HOI Detection with Contextual Cues from Large Vision-Language Models](https://mlsanity.com/paper/2311.16475/) (2023)
- [Geometric Visual Fusion Graph Neural Networks for Multi-Person Human-Object Interaction Recognition in Videos](https://mlsanity.com/paper/2506.03440/) (2025)
- [Egocentric Human-Object Interaction Detection: A New Benchmark and Method](https://mlsanity.com/paper/2506.14189/) (2025)
- [RoHOI: Robustness Benchmark for Human-Object Interaction Detection](https://mlsanity.com/paper/2507.09111/) (2025)
- [Geometric Features Enhanced Human-Object Interaction Detection](https://mlsanity.com/paper/2406.18691/) (2024)
- [Towards Flexible Visual Relationship Segmentation](https://mlsanity.com/paper/2408.08305/) (2024)
- [ContextHOI: Spatial Context Learning for Human-Object Interaction Detection](https://mlsanity.com/paper/2412.09050/) (2024)
- [VLM-HOI: Vision Language Models for Interpretable Human-Object Interaction Analysis](https://mlsanity.com/paper/2411.18038/) (2024)
- [DQEN: Dual Query Enhancement Network for DETR-based HOI Detection](https://mlsanity.com/paper/2508.18896/) (2025)
- [Mining Instance-Centric Vision-Language Contexts for Human-Object Interaction Detection](https://mlsanity.com/paper/2604.02071/) (2026)
- [Unleashing Multimodal Large Language Models for Training-free HOI Detection in the Wild](https://mlsanity.com/paper/2607.13881/) (2026)
- [FreeA: Human-object Interaction Detection using Free Annotation Labels](https://mlsanity.com/paper/2403.01840/) (2024)
- [Contextualized Representation Learning for Effective Human-Object Interaction Detection](https://mlsanity.com/paper/2509.12784/) (2025)

## Abstract (arXiv)

Recent state-of-the-art methods for HOI detection typically build on transformer architectures with two decoder branches, one for human-object pair detection and the other for interaction classification. Such disentangled transformers, however, may suffer from insufficient context exchange between the branches and lead to a lack of context information for relational reasoning, which is critical in discovering HOI instances. In this work, we propose the multiplex relation network (MUREN) that performs rich context exchange between three decoder branches using unary, pairwise, and ternary relations of human, object, and interaction tokens. The proposed method learns comprehensive relational contexts for discovering HOI instances, achieving state-of-the-art performance on two standard benchmarks for HOI detection, HICO-DET and V-COCO.