# Human Interaction Recognition Framework based on Interacting Body Part Attention

arXiv: https://arxiv.org/abs/2101.08967 · HTML: https://mlsanity.com/paper/2101.08967/

## Key facts

- Cited by 11 papers in the map (data through 14 Oct 2026).
- Builds on 23 of its 65 references that are in the map.
- First posted on arXiv 22 Jan 2021.

## Builds on (most cited first, 23 of 23)

- [Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks](https://mlsanity.com/paper/1506.01497/) (2015)
- [Searching for MobileNetV3](https://mlsanity.com/paper/1905.02244/) (2019)
- [Inception-v4, Inception-ResNet and the Impact of Residual Connections on Learning](https://mlsanity.com/paper/1602.07261/) (2016)
- [Deep High-Resolution Representation Learning for Human Pose Estimation](https://mlsanity.com/paper/1902.09212/) (2019)
- [Spatial Temporal Graph Convolutional Networks for Skeleton-Based Action Recognition](https://mlsanity.com/paper/1801.07455/) (2018)
- [OpenPose: Realtime Multi-Person 2D Pose Estimation using Part Affinity Fields](https://mlsanity.com/paper/1812.08008/) (2018)
- [Two-Stream Convolutional Networks for Action Recognition in Videos](https://mlsanity.com/paper/1406.2199/) (2014)
- [NTU RGB+D: A Large Scale Dataset for 3D Human Activity Analysis](https://mlsanity.com/paper/1604.02808/) (2016)
- [Long-term Recurrent Convolutional Networks for Visual Recognition and Description](https://mlsanity.com/paper/1411.4389/) (2014)
- [DeepPose: Human Pose Estimation via Deep Neural Networks](https://mlsanity.com/paper/1312.4659/) (2013)
- [Actional-Structural Graph Convolutional Networks for Skeleton-based Action Recognition](https://mlsanity.com/paper/1904.12659/) (2019)
- [Spatio-Temporal LSTM with Trust Gates for 3D Human Action Recognition](https://mlsanity.com/paper/1607.07043/) (2016)
- [An End-to-End Spatio-Temporal Attention Model for Human Action Recognition from Skeleton Data](https://mlsanity.com/paper/1611.06067/) (2016)
- [2D/3D Pose Estimation and Action Recognition using Multitask Deep Learning](https://mlsanity.com/paper/1802.09232/) (2018)
- [Hierarchical Deep Temporal Models for Group Activity Recognition](https://mlsanity.com/paper/1607.02643/) (2016)
- [Skeleton-Based Human Action Recognition with Global Context-Aware Attention LSTM Networks](https://mlsanity.com/paper/1707.05740/) (2017)
- [Hierarchical Long Short-Term Concurrent Memory for Human Interaction Recognition](https://mlsanity.com/paper/1811.00270/) (2018)
- [Structure Inference Machines: Recurrent Neural Networks for Analyzing Relations in Group Activity Recognition](https://mlsanity.com/paper/1511.04196/) (2015)
- [Interaction Relational Network for Mutual Action Recognition](https://mlsanity.com/paper/1910.04963/) (2019)
- [Feedback Graph Convolutional Network for Skeleton-based Action Recognition](https://mlsanity.com/paper/2003.07564/) (2020)
- [Human Action Recognition: Pose-based Attention draws focus to Hands](https://mlsanity.com/paper/1712.08002/) (2017)
- [Skeleton-Based Online Action Prediction Using Scale Selection Network](https://mlsanity.com/paper/1902.03084/) (2019)
- [Concurrence-Aware Long Short-Term Sub-Memories for Person-Person Action Recognition](https://mlsanity.com/paper/1706.00931/) (2017)

## Built on by (most cited first, 11 of 11)

- [Music-Driven Group Choreography](https://mlsanity.com/paper/2303.12337/) (2023)
- [JRDB-Social: A Multifaceted Robotic Dataset for Understanding of Context and Dynamics of Human Interactions Within Social Groups](https://mlsanity.com/paper/2404.04458/) (2024)
- [Multi-Contextual Predictions with Vision Transformer for Video Anomaly Detection](https://mlsanity.com/paper/2206.08568/) (2022)
- [HTNet: Anchor-free Temporal Action Localization with Hierarchical Transformers](https://mlsanity.com/paper/2207.09662/) (2022)
- [Learning Human-Human Interactions in Images from Weak Textual Supervision](https://mlsanity.com/paper/2304.14104/) (2023)
- [AdaFPP: Adapt-Focused Bi-Propagating Prototype Learning for Panoramic Activity Recognition](https://mlsanity.com/paper/2405.02538/) (2024)
- [Few-Shot Object Detection with Proposal Balance Refinement](https://mlsanity.com/paper/2204.10527/) (2022)
- [Factorization Approach for Sparse Spatio-Temporal Brain-Computer Interface](https://mlsanity.com/paper/2206.08494/) (2022)
- [Rewarded meta-pruning: Meta Learning with Rewards for Channel Pruning](https://mlsanity.com/paper/2301.11063/) (2023)
- [Structure over Depth: A Single-Block Spatio-Temporal Transformer for Multi-Entity Reasoning](https://mlsanity.com/paper/2607.23077/) (2026)
- [BeatDance: Generating Beat-Consistent 3D Dance with Hierarchical Spatial-Temporal Modeling](https://mlsanity.com/paper/2609.37400/) (2026)

## Abstract (arXiv)

Human activity recognition in videos has been widely studied and has recently gained significant advances with deep learning approaches; however, it remains a challenging task. In this paper, we propose a novel framework that simultaneously considers both implicit and explicit representations of human interactions by fusing information of local image where the interaction actively occurred, primitive motion with the posture of individual subject's body parts, and the co-occurrence of overall appearance change. Human interactions change, depending on how the body parts of each human interact with the other. The proposed method captures the subtle difference between different interactions using interacting body part attention. Semantically important body parts that interact with other objects are given more weight during feature representation. The combined feature of interacting body part attention-based individual representation and the co-occurrence descriptor of the full-body appearance change is fed into long short-term memory to model the temporal dynamics over time in a single framework. We validate the effectiveness of the proposed method using four widely used public datasets by outperforming the competing state-of-the-art method.