# Pairwise Body-Part Attention for Recognizing Human-Object Interactions

arXiv: https://arxiv.org/abs/1807.10889 · HTML: https://mlsanity.com/paper/1807.10889/

## Key facts

- Cited by 34 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 28 Jul 2018.

## Built on by (most cited first, 30 of 34)

- [Observation-Centric SORT: Rethinking SORT for Robust Multi-Object Tracking](https://mlsanity.com/paper/2203.14360/) (2022)
- [H2O: Two Hands Manipulating Objects for First Person Interaction Recognition](https://mlsanity.com/paper/2104.11181/) (2021)
- [DanceTrack: Multi-Object Tracking in Uniform Appearance and Diverse Motion](https://mlsanity.com/paper/2111.14690/) (2021)
- [Recent Advances in Monocular 2D and 3D Human Pose Estimation: A Deep Learning Perspective](https://mlsanity.com/paper/2104.11536/) (2021)
- [Mining Cross-Person Cues for Body-Part Interactiveness Learning in HOI Detection](https://mlsanity.com/paper/2207.14192/) (2022)
- [UnionDet: Union-Level Detector Towards Real-Time Human-Object Interaction Detection](https://mlsanity.com/paper/2312.12664/) (2023)
- [HAKE: A Knowledge Engine Foundation for Human Activity Understanding](https://mlsanity.com/paper/2202.06851/) (2022)
- [Category-Aware Transformer Network for Better Human-Object Interaction Detection](https://mlsanity.com/paper/2204.04911/) (2022)
- [From Isolated Islands to Pangea: Unifying Semantic Space for Human Action Understanding](https://mlsanity.com/paper/2304.00553/) (2023)
- [Geometric Features Informed Multi-person Human-object Interaction Recognition in Videos](https://mlsanity.com/paper/2207.09425/) (2022)
- [Symbol-LLM: Leverage Language Models for Symbolic System in Visual Human Activity Reasoning](https://mlsanity.com/paper/2311.17365/) (2023)
- [Improving Human-Object Interaction Detection via Phrase Learning and Label Composition](https://mlsanity.com/paper/2112.07383/) (2021)
- [RelViT: Concept-guided Vision Transformer for Visual Relational Reasoning](https://mlsanity.com/paper/2204.11167/) (2022)
- [Video Action Recognition with Attentive Semantic Units](https://mlsanity.com/paper/2303.09756/) (2023)
- [Align before Adapt: Leveraging Entity-to-Region Alignments for Generalizable Video Action Recognition](https://mlsanity.com/paper/2311.15619/) (2023)
- [Highlighting Object Category Immunity for the Generalization of Human-Object Interaction Detection](https://mlsanity.com/paper/2202.09492/) (2022)
- [GTNet:Guided Transformer Network for Detecting Human-Object Interactions](https://mlsanity.com/paper/2108.00596/) (2021)
- [The Overlooked Classifier in Human-Object Interaction Recognition](https://mlsanity.com/paper/2112.06392/) (2021)
- [An Image-like Diffusion Method for Human-Object Interaction Detection](https://mlsanity.com/paper/2503.18134/) (2025)
- [Human Object Interaction Detection using Two-Direction Spatial Enhancement and Exclusive Object Prior](https://mlsanity.com/paper/2105.03089/) (2021)
- [Is Object Detection Necessary for Human-Object Interaction Recognition?](https://mlsanity.com/paper/2107.13083/) (2021)
- [RoHOI: Robustness Benchmark for Human-Object Interaction Detection](https://mlsanity.com/paper/2507.09111/) (2025)
- [HOI-aware Adaptive Network for Weakly-supervised Action Segmentation](https://mlsanity.com/paper/2604.26227/) (2026)
- [Geometric Features Enhanced Human-Object Interaction Detection](https://mlsanity.com/paper/2406.18691/) (2024)
- [Joint Image-Instance Spatial-Temporal Attention for Few-shot Action Recognition](https://mlsanity.com/paper/2503.14430/) (2025)
- [HERO-VQL: Hierarchical, Egocentric and Robust Visual Query Localization](https://mlsanity.com/paper/2509.00385/) (2025)
- [Transferable Interactiveness Knowledge for Human-Object Interaction Detection](https://mlsanity.com/paper/2101.10292/) (2021)
- [RR-Net: Injecting Interactive Semantics in Human-Object Interaction Detection](https://mlsanity.com/paper/2104.15015/) (2021)
- [Intention Action Anticipation Model with Guide-Feedback Loop Mechanism](https://mlsanity.com/paper/2403.12450/) (2024)
- [CL-HOI: Cross-Level Human-Object Interaction Distillation from Vision Large Language Models](https://mlsanity.com/paper/2410.15657/) (2024)

## Abstract (arXiv)

In human-object interactions (HOI) recognition, conventional methods consider the human body as a whole and pay a uniform attention to the entire body region. They ignore the fact that normally, human interacts with an object by using some parts of the body. In this paper, we argue that different body parts should be paid with different attention in HOI recognition, and the correlations between different body parts should be further considered. This is because our body parts always work collaboratively. We propose a new pairwise body-part attention model which can learn to focus on crucial parts, and their correlations for HOI recognition. A novel attention based feature selection method and a feature representation scheme that can capture pairwise correlations between body parts are introduced in the model. Our proposed approach achieved 4% improvement over the state-of-the-art results in HOI recognition on the HICO dataset. We will make our model and source codes publicly available.