# Cascaded Human-Object Interaction Recognition

arXiv: https://arxiv.org/abs/2003.04262 · HTML: https://mlsanity.com/paper/2003.04262/

## Key facts

- Cited by 40 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 9 Mar 2020.

## Built on by (most cited first, 30 of 40)

- [QPIC: Query-Based Pairwise Human-Object Interaction Detection with Image-Wide Contextual Information](https://mlsanity.com/paper/2103.05399/) (2021)
- [End-to-End Human Object Interaction Detection with HOI Transformer](https://mlsanity.com/paper/2103.04503/) (2021)
- [Panoptic Scene Graph Generation](https://mlsanity.com/paper/2207.11247/) (2022)
- [Reformulating HOI Detection as Adaptive Set Prediction](https://mlsanity.com/paper/2103.05983/) (2021)
- [CLIP-Event: Connecting Text and Images with Event Structures](https://mlsanity.com/paper/2201.05078/) (2022)
- [Open-Vocabulary Instance Segmentation via Robust Cross-Modal Pseudo-Labeling](https://mlsanity.com/paper/2111.12698/) (2021)
- [Glance and Gaze: Inferring Action-aware Points for One-Stage Human-Object Interaction Detection](https://mlsanity.com/paper/2104.05269/) (2021)
- [Neural-Logic Human-Object Interaction Detection](https://mlsanity.com/paper/2311.09817/) (2023)
- [Iwin: Human-Object Interaction Detection via Transformer with Irregular Windows](https://mlsanity.com/paper/2203.10537/) (2022)
- [GaTector: A Unified Framework for Gaze Object Prediction](https://mlsanity.com/paper/2112.03549/) (2021)
- [Fashionformer: A simple, Effective and Unified Baseline for Human Fashion Segmentation and Recognition](https://mlsanity.com/paper/2204.04654/) (2022)
- [Improving Human-Object Interaction Detection via Phrase Learning and Label Composition](https://mlsanity.com/paper/2112.07383/) (2021)
- [Human-Object Interaction Detection Collaborated with Large Relation-driven Diffusion Models](https://mlsanity.com/paper/2410.20155/) (2024)
- [Video-based Human-Object Interaction Detection from Tubelet Tokens](https://mlsanity.com/paper/2206.01908/) (2022)
- [Towards Hard-Positive Query Mining for DETR-based Human-Object Interaction Detection](https://mlsanity.com/paper/2207.05293/) (2022)
- [Weakly-supervised HOI Detection via Prior-guided Bi-level Representation Learning](https://mlsanity.com/paper/2303.01313/) (2023)
- [DIFFVSGG: Diffusion-Driven Online Video Scene Graph Generation](https://mlsanity.com/paper/2503.13957/) (2025)
- [Hydra-SGG: Hybrid Relation Assignment for One-stage Scene Graph Generation](https://mlsanity.com/paper/2409.10262/) (2024)
- [Agglomerative Transformer for Human-Object Interaction Detection](https://mlsanity.com/paper/2308.08370/) (2023)
- [Human-centric Relation Segmentation: Dataset and Solution](https://mlsanity.com/paper/2105.11168/) (2021)
- [Self-supervised Social Relation Representation for Human Group Detection](https://mlsanity.com/paper/2203.03843/) (2022)
- [Video-based Contrastive Learning on Decision Trees: from Action Recognition to Autism Diagnosis](https://mlsanity.com/paper/2304.10073/) (2023)
- [Exploiting CLIP for Zero-shot HOI Detection Requires Knowledge Distillation at Multiple Levels](https://mlsanity.com/paper/2309.05069/) (2023)
- [Learning Human-Object Interaction as Groups](https://mlsanity.com/paper/2510.18357/) (2025)
- [HAtt-Flow: Hierarchical Attention-Flow Mechanism for Group Activity Scene Graph Generation in Videos](https://mlsanity.com/paper/2312.07740/) (2023)
- [UAHOI: Uncertainty-aware Robust Interaction Learning for HOI Detection](https://mlsanity.com/paper/2408.07430/) (2024)
- [OmniGaze: Reward-inspired Generalizable Gaze Estimation In The Wild](https://mlsanity.com/paper/2510.13660/) (2025)
- [Frequency-based Matcher for Long-tailed Semantic Segmentation](https://mlsanity.com/paper/2406.03917/) (2024)
- [RoHOI: Robustness Benchmark for Human-Object Interaction Detection](https://mlsanity.com/paper/2507.09111/) (2025)
- [No More Sibling Rivalry: Debiasing Human-Object Interaction Detection](https://mlsanity.com/paper/2509.00760/) (2025)

## Abstract (arXiv)

Rapid progress has been witnessed for human-object interaction (HOI) recognition, but most existing models are confined to single-stage reasoning pipelines. Considering the intrinsic complexity of the task, we introduce a cascade architecture for a multi-stage, coarse-to-fine HOI understanding. At each stage, an instance localization network progressively refines HOI proposals and feeds them into an interaction recognition network. Each of the two networks is also connected to its predecessor at the previous stage, enabling cross-stage information propagation. The interaction recognition network has two crucial parts: a relation ranking module for high-quality HOI proposal selection and a triple-stream classifier for relation prediction. With our carefully-designed human-centric relation features, these two modules work collaboratively towards effective interaction understanding. Further beyond relation detection on a bounding-box level, we make our framework flexible to perform fine-grained pixel-wise relation segmentation; this provides a new glimpse into better relation modeling. Our approach reached the 1st place in the ICCV2019 Person in Context Challenge, on both relation detection and segmentation tasks. It also shows promising results on V-COCO.