# Spatio-Temporal Interaction Graph Parsing Networks for Human-Object Interaction Recognition

arXiv: https://arxiv.org/abs/2108.08633 · HTML: https://mlsanity.com/paper/2108.08633/

## Key facts

- Cited by 12 papers in the map (data through 14 Oct 2026).
- Builds on 28 of its 44 references that are in the map.
- First posted on arXiv 19 Aug 2021.

## Builds on (most cited first, 28 of 28)

- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Microsoft COCO: Common Objects in Context](https://mlsanity.com/paper/1405.0312/) (2014)
- [Semi-Supervised Classification with Graph Convolutional Networks](https://mlsanity.com/paper/1609.02907/) (2016)
- [Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks](https://mlsanity.com/paper/1506.01497/) (2015)
- [You Only Look Once: Unified, Real-Time Object Detection](https://mlsanity.com/paper/1506.02640/) (2015)
- [Spatial Temporal Graph Convolutional Networks for Skeleton-Based Action Recognition](https://mlsanity.com/paper/1801.07455/) (2018)
- [The "something something" video database for learning and evaluating visual common sense](https://mlsanity.com/paper/1706.04261/) (2017)
- [Convolutional Two-Stream Network Fusion for Video Action Recognition](https://mlsanity.com/paper/1604.06573/) (2016)
- [Relation Networks for Object Detection](https://mlsanity.com/paper/1711.11575/) (2017)
- [Deep Graph Library: A Graph-Centric, Highly-Performant Package for Graph Neural Networks](https://mlsanity.com/paper/1909.01315/) (2019)
- [Structural-RNN: Deep Learning on Spatio-Temporal Graphs](https://mlsanity.com/paper/1511.05298/) (2015)
- [Videos as Space-Time Region Graphs](https://mlsanity.com/paper/1806.01810/) (2018)
- [Detecting and Recognizing Human-Object Interactions](https://mlsanity.com/paper/1704.07333/) (2017)
- [Learning Human-Object Interactions by Graph Parsing Neural Networks](https://mlsanity.com/paper/1808.07962/) (2018)
- [An Attention Enhanced Graph Convolutional LSTM Network for Skeleton-Based Action Recognition](https://mlsanity.com/paper/1902.09130/) (2019)
- [iCAN: Instance-Centric Attention Network for Human-Object Interaction Detection](https://mlsanity.com/paper/1808.10437/) (2018)
- [DRG: Dual Relation Graph for Human-Object Interaction Detection](https://mlsanity.com/paper/2008.11714/) (2020)
- [Something-Else: Compositional Action Recognition with Spatial-Temporal Interaction Networks](https://mlsanity.com/paper/1912.09930/) (2019)
- [Skeleton-Based Action Recognition with Multi-Stream Adaptive Graph Convolutional Networks](https://mlsanity.com/paper/1912.06971/) (2019)
- [Video Relationship Reasoning using Gated Spatio-Temporal Energy Graph](https://mlsanity.com/paper/1903.10547/) (2019)
- [Scalable Object Detection using Deep Neural Networks](https://mlsanity.com/paper/1312.2249/) (2013)
- [Interact as You Intend: Intention-Driven Human-Object Interaction Detection](https://mlsanity.com/paper/1808.09796/) (2018)
- [LIGHTEN: Learning Interactions with Graph and Hierarchical TEmporal Networks for HOI in videos](https://mlsanity.com/paper/2012.09402/) (2020)
- [Transferable Interactiveness Knowledge for Human-Object Interaction Detection](https://mlsanity.com/paper/1811.08264/) (2018)
- [Scalable, High-Quality Object Detection](https://mlsanity.com/paper/1412.1441/) (2014)
- [Reasoning About Human-Object Interactions Through Dual Attention Networks](https://mlsanity.com/paper/1909.04743/) (2019)
- [Attention-based Temporal Weighted Convolutional Neural Network for Action Recognition](https://mlsanity.com/paper/1803.07179/) (2018)
- [Pose-based Modular Network for Human-Object Interaction Detection](https://mlsanity.com/paper/2008.02042/) (2020)

## Built on by (most cited first, 12 of 12)

- [Geometric Features Informed Multi-person Human-object Interaction Recognition in Videos](https://mlsanity.com/paper/2207.09425/) (2022)
- [Video-based Human-Object Interaction Detection from Tubelet Tokens](https://mlsanity.com/paper/2206.01908/) (2022)
- [Human-Object Interaction Prediction in Videos through Gaze Following](https://mlsanity.com/paper/2306.03597/) (2023)
- [MUSEG: Reinforcing Video Temporal Understanding via Timestamp-Aware Multi-Segment Grounding](https://mlsanity.com/paper/2505.20715/) (2025)
- [Enhancing Human-Centered Dynamic Scene Understanding via Multiple LLMs Collaborated Reasoning](https://mlsanity.com/paper/2403.10107/) (2024)
- [From Category to Scenery: An End-to-End Framework for Multi-Person Human-Object Interaction Recognition in Videos](https://mlsanity.com/paper/2407.00917/) (2024)
- [Dynamic Compositional Graph Convolutional Network for Efficient Composite Human Motion Prediction](https://mlsanity.com/paper/2311.13781/) (2023)
- [Geometric Visual Fusion Graph Neural Networks for Multi-Person Human-Object Interaction Recognition in Videos](https://mlsanity.com/paper/2506.03440/) (2025)
- [Rethinking Video Human-Object Interaction: Set Prediction over Time for Unified Detection and Anticipation](https://mlsanity.com/paper/2604.10397/) (2026)
- [IMPACT-HOI: Supervisory Control for Onset-Anchored Partial HOI Event Construction](https://mlsanity.com/paper/2605.01666/) (2026)
- [Language-Driven Object-Oriented Two-Stage Method for Scene Graph Anticipation](https://mlsanity.com/paper/2509.05661/) (2025)
- [Group Activity Recognition in Computer Vision: A Comprehensive Review, Challenges, and Future Perspectives](https://mlsanity.com/paper/2307.13541/) (2023)

## Abstract (arXiv)

For a given video-based Human-Object Interaction scene, modeling the spatio-temporal relationship between humans and objects are the important cue to understand the contextual information presented in the video. With the effective spatio-temporal relationship modeling, it is possible not only to uncover contextual information in each frame but also to directly capture inter-time dependencies. It is more critical to capture the position changes of human and objects over the spatio-temporal dimension when their appearance features may not show up significant changes over time. The full use of appearance features, the spatial location and the semantic information are also the key to improve the video-based Human-Object Interaction recognition performance. In this paper, Spatio-Temporal Interaction Graph Parsing Networks (STIGPN) are constructed, which encode the videos with a graph composed of human and object nodes. These nodes are connected by two types of relations: (i) spatial relations modeling the interactions between human and the interacted objects within each frame. (ii) inter-time relations capturing the long range dependencies between human and the interacted objects across frame. With the graph, STIGPN learn spatio-temporal features directly from the whole video-based Human-Object Interaction scenes. Multi-modal features and a multi-stream fusion strategy are used to enhance the reasoning capability of STIGPN. Two Human-Object Interaction video datasets, including CAD-120 and Something-Else, are used to evaluate the proposed architectures, and the state-of-the-art performance demonstrates the superiority of STIGPN.