# Polysemy Deciphering Network for Robust Human-Object Interaction Detection

arXiv: https://arxiv.org/abs/2008.02918 · HTML: https://mlsanity.com/paper/2008.02918/

## Key facts

- Cited by 25 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 7 Aug 2020.

## Built on by (most cited first, 25 of 25)

- [QPIC: Query-Based Pairwise Human-Object Interaction Detection with Image-Wide Contextual Information](https://mlsanity.com/paper/2103.05399/) (2021)
- [Mining the Benefits of Two-stage and One-stage HOI Detection](https://mlsanity.com/paper/2108.05077/) (2021)
- [Affordance Transfer Learning for Human-Object Interaction Detection](https://mlsanity.com/paper/2104.02867/) (2021)
- [Glance and Gaze: Inferring Action-aware Points for One-Stage Human-Object Interaction Detection](https://mlsanity.com/paper/2104.05269/) (2021)
- [One-Shot Object Affordance Detection in the Wild](https://mlsanity.com/paper/2108.03658/) (2021)
- [Detecting Any Human-Object Interaction Relationship: Universal HOI Detector with Spatial Prompt Learning on Foundation Models](https://mlsanity.com/paper/2311.03799/) (2023)
- [Visual Relationship Detection Using Part-and-Sum Transformers with Composite Queries](https://mlsanity.com/paper/2105.02170/) (2021)
- [FGAHOI: Fine-Grained Anchors for Human-Object Interaction Detection](https://mlsanity.com/paper/2301.04019/) (2023)
- [QAHOI: Query-Based Anchors for Human-Object Interaction Detection](https://mlsanity.com/paper/2112.08647/) (2021)
- [Category-Aware Transformer Network for Better Human-Object Interaction Detection](https://mlsanity.com/paper/2204.04911/) (2022)
- [Category Query Learning for Human-Object Interaction Classification](https://mlsanity.com/paper/2303.14005/) (2023)
- [Iwin: Human-Object Interaction Detection via Transformer with Irregular Windows](https://mlsanity.com/paper/2203.10537/) (2022)
- [Consistency Learning via Decoding Path Augmentation for Transformers in Human Object Interaction Detection](https://mlsanity.com/paper/2204.04836/) (2022)
- [Video-based Human-Object Interaction Detection from Tubelet Tokens](https://mlsanity.com/paper/2206.01908/) (2022)
- [Re-mine, Learn and Reason: Exploring the Cross-modal Semantic Correlations for Language-guided HOI detection](https://mlsanity.com/paper/2307.13529/) (2023)
- [Towards Hard-Positive Query Mining for DETR-based Human-Object Interaction Detection](https://mlsanity.com/paper/2207.05293/) (2022)
- [Agglomerative Transformer for Human-Object Interaction Detection](https://mlsanity.com/paper/2308.08370/) (2023)
- [Focusing on what to decode and what to train: SOV Decoding with Specific Target Guided DeNoising and Vision Language Advisor](https://mlsanity.com/paper/2307.02291/) (2023)
- [Enhancing HOI Detection with Contextual Cues from Large Vision-Language Models](https://mlsanity.com/paper/2311.16475/) (2023)
- [Streamlined Open-Vocabulary Human-Object Interaction Detection](https://mlsanity.com/paper/2603.27500/) (2026)
- [An Action Is Worth Multiple Words: Handling Ambiguity in Action Recognition](https://mlsanity.com/paper/2210.04933/) (2022)
- [TMHOI: Translational Model for Human-Object Interaction Detection](https://mlsanity.com/paper/2303.04253/) (2023)
- [Mining Conditional Part Semantics with Occluded Extrapolation for Human-Object Interaction Detection](https://mlsanity.com/paper/2307.10499/) (2023)
- [Disentangled Interaction Representation for One-Stage Human-Object Interaction Detection](https://mlsanity.com/paper/2312.01713/) (2023)
- [Efficient Human-Object-Interaction (EHOI) Detection via Interaction Label Coding and Conditional Decision](https://mlsanity.com/paper/2408.07018/) (2024)

## Abstract (arXiv)

Human-Object Interaction (HOI) detection is important to human-centric scene understanding tasks. Existing works tend to assume that the same verb has similar visual characteristics in different HOI categories, an approach that ignores the diverse semantic meanings of the verb. To address this issue, in this paper, we propose a novel Polysemy Deciphering Network (PD-Net) that decodes the visual polysemy of verbs for HOI detection in three distinct ways. First, we refine features for HOI detection to be polysemyaware through the use of two novel modules: namely, Language Prior-guided Channel Attention (LPCA) and Language Prior-based Feature Augmentation (LPFA). LPCA highlights important elements in human and object appearance features for each HOI category to be identified; moreover, LPFA augments human pose and spatial features for HOI detection using language priors, enabling the verb classifiers to receive language hints that reduce intra-class variation for the same verb. Second, we introduce a novel Polysemy-Aware Modal Fusion module (PAMF), which guides PD-Net to make decisions based on feature types deemed more important according to the language priors. Third, we propose to relieve the verb polysemy problem through sharing verb classifiers for semantically similar HOI categories. Furthermore, to expedite research on the verb polysemy problem, we build a new benchmark dataset named HOI-VerbPolysemy (HOIVP), which includes common verbs (predicates) that have diverse semantic meanings in the real world. Finally, through deciphering the visual polysemy of verbs, our approach is demonstrated to outperform state-of-the-art methods by significant margins on the HICO-DET, V-COCO, and HOI-VP databases. Code and data in this paper are available at https://github.com/MuchHair/PD-Net.