# UnionDet: Union-Level Detector Towards Real-Time Human-Object Interaction Detection

arXiv: https://arxiv.org/abs/2312.12664 · HTML: https://mlsanity.com/paper/2312.12664/

## Key facts

- Cited by 31 papers in the map (data through 14 Oct 2026).
- Builds on 26 of its 35 references that are in the map.
- First posted on arXiv 19 Dec 2023.

## Builds on (most cited first, 26 of 26)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks](https://mlsanity.com/paper/1506.01497/) (2015)
- [Feature Pyramid Networks for Object Detection](https://mlsanity.com/paper/1612.03144/) (2016)
- [You Only Look Once: Unified, Real-Time Object Detection](https://mlsanity.com/paper/1506.02640/) (2015)
- [Distributed Representations of Words and Phrases and their Compositionality](https://mlsanity.com/paper/1310.4546/) (2013)
- [SSD: Single Shot MultiBox Detector](https://mlsanity.com/paper/1512.02325/) (2015)
- [Realtime Multi-Person 2D Pose Estimation using Part Affinity Fields](https://mlsanity.com/paper/1611.08050/) (2016)
- [YOLACT: Real-time Instance Segmentation](https://mlsanity.com/paper/1904.02689/) (2019)
- [Cascaded Pyramid Network for Multi-Person Pose Estimation](https://mlsanity.com/paper/1711.07319/) (2017)
- [RMPE: Regional Multi-person Pose Estimation](https://mlsanity.com/paper/1612.00137/) (2016)
- [Visual Relationship Detection with Language Priors](https://mlsanity.com/paper/1608.00187/) (2016)
- [Learning to Detect Human-Object Interactions](https://mlsanity.com/paper/1702.05448/) (2017)
- [OverFeat: Integrated Recognition, Localization and Detection using Convolutional Networks](https://mlsanity.com/paper/1312.6229/) (2013)
- [CrowdPose: Efficient Crowded Scenes Pose Estimation and A New Benchmark](https://mlsanity.com/paper/1812.00324/) (2018)
- [Detecting and Recognizing Human-Object Interactions](https://mlsanity.com/paper/1704.07333/) (2017)
- [Visual Semantic Role Labeling](https://mlsanity.com/paper/1505.04474/) (2015)
- [Learning Human-Object Interactions by Graph Parsing Neural Networks](https://mlsanity.com/paper/1808.07962/) (2018)
- [Single-Shot Refinement Neural Network for Object Detection](https://mlsanity.com/paper/1711.06897/) (2017)
- [iCAN: Instance-Centric Attention Network for Human-Object Interaction Detection](https://mlsanity.com/paper/1808.10437/) (2018)
- [Pose-aware Multi-level Feature Network for Human Object Interaction Detection](https://mlsanity.com/paper/1909.08453/) (2019)
- [M2Det: A Single-Shot Object Detector based on Multi-Level Feature Pyramid Network](https://mlsanity.com/paper/1811.04533/) (2018)
- [Deep Contextual Attention for Human-Object Interaction Detection](https://mlsanity.com/paper/1910.07721/) (2019)
- [Detecting Human-Object Interactions via Functional Generalization](https://mlsanity.com/paper/1904.03181/) (2019)
- [Pairwise Body-Part Attention for Recognizing Human-Object Interactions](https://mlsanity.com/paper/1807.10889/) (2018)
- [Detecting Visual Relationships Using Box Attention](https://mlsanity.com/paper/1807.02136/) (2018)
- [Aligned Image-Word Representations Improve Inductive Transfer Across Vision-Language Tasks](https://mlsanity.com/paper/1704.00260/) (2017)

## Built on by (most cited first, 30 of 31)

- [Open-World Human-Object Interaction Detection via Multi-modal Prompts](https://mlsanity.com/paper/2406.07221/) (2024)
- [Exploring the Potential of Large Foundation Models for Open-Vocabulary HOI Detection](https://mlsanity.com/paper/2404.06194/) (2024)
- [Learning from Observer Gaze:Zero-Shot Attention Prediction Oriented by Human-Object Interaction Recognition](https://mlsanity.com/paper/2405.09931/) (2024)
- [Human-Object Interaction Detection Collaborated with Large Relation-driven Diffusion Models](https://mlsanity.com/paper/2410.20155/) (2024)
- [Nonverbal Interaction Detection](https://mlsanity.com/paper/2407.08133/) (2024)
- [Locality-Aware Zero-Shot Human-Object Interaction Detection](https://mlsanity.com/paper/2505.19503/) (2025)
- [Groupwise Query Specialization and Quality-Aware Multi-Assignment for Transformer-based Visual Relationship Detection](https://mlsanity.com/paper/2403.17709/) (2024)
- [Open-Vocabulary HOI Detection with Interaction-aware Prompt and Concept Calibration](https://mlsanity.com/paper/2508.03207/) (2025)
- [Exploring Self- and Cross-Triplet Correlations for Human-Object Interaction Detection](https://mlsanity.com/paper/2401.05676/) (2024)
- [Enhancing Human-Centered Dynamic Scene Understanding via Multiple LLMs Collaborated Reasoning](https://mlsanity.com/paper/2403.10107/) (2024)
- [A Review of Human-Object Interaction Detection](https://mlsanity.com/paper/2408.10641/) (2024)
- [An Image-like Diffusion Method for Human-Object Interaction Detection](https://mlsanity.com/paper/2503.18134/) (2025)
- [Learning Human-Object Interaction as Groups](https://mlsanity.com/paper/2510.18357/) (2025)
- [UAHOI: Uncertainty-aware Robust Interaction Learning for HOI Detection](https://mlsanity.com/paper/2408.07430/) (2024)
- [SGC-Net: Stratified Granular Comparison Network for Open-Vocabulary HOI Detection](https://mlsanity.com/paper/2503.00414/) (2025)
- [Egocentric Human-Object Interaction Detection: A New Benchmark and Method](https://mlsanity.com/paper/2506.14189/) (2025)
- [RoHOI: Robustness Benchmark for Human-Object Interaction Detection](https://mlsanity.com/paper/2507.09111/) (2025)
- [No More Sibling Rivalry: Debiasing Human-Object Interaction Detection](https://mlsanity.com/paper/2509.00760/) (2025)
- [Visual Diversity and Region-aware Prompt Learning for Zero-shot HOI Detection](https://mlsanity.com/paper/2510.25094/) (2025)
- [HOI-aware Adaptive Network for Weakly-supervised Action Segmentation](https://mlsanity.com/paper/2604.26227/) (2026)
- [Geometric Features Enhanced Human-Object Interaction Detection](https://mlsanity.com/paper/2406.18691/) (2024)
- [Towards Flexible Visual Relationship Segmentation](https://mlsanity.com/paper/2408.08305/) (2024)
- [Dynamic Scoring with Enhanced Semantics for Training-Free Human-Object Interaction Detection](https://mlsanity.com/paper/2507.17456/) (2025)
- [HOID-R1: Reinforcement Learning for Open-World Human-Object Interaction Detection Reasoning with Multimodal Large Language Model](https://mlsanity.com/paper/2508.11350/) (2025)
- [SGTR+: End-to-end Scene Graph Generation with Transformer](https://mlsanity.com/paper/2401.12835/) (2024)
- [Mining Instance-Centric Vision-Language Contexts for Human-Object Interaction Detection](https://mlsanity.com/paper/2604.02071/) (2026)
- [On the Robustness of Human-Object Interaction Detection against Distribution Shift](https://mlsanity.com/paper/2506.18021/) (2025)
- [Conceptualizing Multi-scale Wavelet Attention and Ray-based Encoding for Human-Object Interaction Detection](https://mlsanity.com/paper/2507.10977/) (2025)
- [Multimodal Graph Network Modeling for Human-Object Interaction Detection with PDE Graph Diffusion](https://mlsanity.com/paper/2509.12554/) (2025)
- [RegFormer: Transferable Relational Grounding for Efficient Weakly-Supervised Human-Object Interaction Detection](https://mlsanity.com/paper/2604.00507/) (2026)

## Abstract (arXiv)

Recent advances in deep neural networks have achieved significant progress in detecting individual objects from an image. However, object detection is not sufficient to fully understand a visual scene. Towards a deeper visual understanding, the interactions between objects, especially humans and objects are essential. Most prior works have obtained this information with a bottom-up approach, where the objects are first detected and the interactions are predicted sequentially by pairing the objects. This is a major bottleneck in HOI detection inference time. To tackle this problem, we propose UnionDet, a one-stage meta-architecture for HOI detection powered by a novel union-level detector that eliminates this additional inference stage by directly capturing the region of interaction. Our one-stage detector for human-object interaction shows a significant reduction in interaction prediction time 4x~14x while outperforming state-of-the-art methods on two public datasets: V-COCO and HICO-DET.