# Improving Human-Object Interaction Detection via Phrase Learning and Label Composition

arXiv: https://arxiv.org/abs/2112.07383 · HTML: https://mlsanity.com/paper/2112.07383/

## Key facts

- Cited by 15 papers in the map (data through 14 Oct 2026).
- Builds on 26 of its 36 references that are in the map.
- First posted on arXiv 14 Dec 2021.

## Builds on (most cited first, 26 of 26)

- [End-to-End Object Detection with Transformers](https://mlsanity.com/paper/2005.12872/) (2020)
- [Distributed Representations of Words and Phrases and their Compositionality](https://mlsanity.com/paper/1310.4546/) (2013)
- [Distributed Representations of Sentences and Documents](https://mlsanity.com/paper/1405.4053/) (2014)
- [Visual Relationship Detection with Language Priors](https://mlsanity.com/paper/1608.00187/) (2016)
- [Learning to Detect Human-Object Interactions](https://mlsanity.com/paper/1702.05448/) (2017)
- [Visual Semantic Role Labeling](https://mlsanity.com/paper/1505.04474/) (2015)
- [HOTR: End-to-End Human-Object Interaction Detection with Transformers](https://mlsanity.com/paper/2104.13682/) (2021)
- [Cross Modal Distillation for Supervision Transfer](https://mlsanity.com/paper/1507.00448/) (2015)
- [QPIC: Query-Based Pairwise Human-Object Interaction Detection with Image-Wide Contextual Information](https://mlsanity.com/paper/2103.05399/) (2021)
- [PPDM: Parallel Point Detection and Matching for Real-time Human-Object Interaction Detection](https://mlsanity.com/paper/1912.12898/) (2019)
- [End-to-End Human Object Interaction Detection with HOI Transformer](https://mlsanity.com/paper/2103.04503/) (2021)
- [Visual Compositional Learning for Human-Object Interaction Detection](https://mlsanity.com/paper/2007.12407/) (2020)
- [iCAN: Instance-Centric Attention Network for Human-Object Interaction Detection](https://mlsanity.com/paper/1808.10437/) (2018)
- [DRG: Dual Relation Graph for Human-Object Interaction Detection](https://mlsanity.com/paper/2008.11714/) (2020)
- [VSGNet: Spatial Attention Network for Detecting Human Object Interactions Using Graph Convolutions](https://mlsanity.com/paper/2003.05541/) (2020)
- [Learning Human-Object Interaction Detection using Interaction Points](https://mlsanity.com/paper/2003.14023/) (2020)
- [Mining the Benefits of Two-stage and One-stage HOI Detection](https://mlsanity.com/paper/2108.05077/) (2021)
- [Reformulating HOI Detection as Adaptive Set Prediction](https://mlsanity.com/paper/2103.05983/) (2021)
- [Pose-aware Multi-level Feature Network for Human Object Interaction Detection](https://mlsanity.com/paper/1909.08453/) (2019)
- [PaStaNet: Toward Human Activity Knowledge Engine](https://mlsanity.com/paper/2004.00945/) (2020)
- [Detecting Human-Object Interactions with Action Co-occurrence Priors](https://mlsanity.com/paper/2007.08728/) (2020)
- [Deep Contextual Attention for Human-Object Interaction Detection](https://mlsanity.com/paper/1910.07721/) (2019)
- [Detecting Human-Object Interactions via Functional Generalization](https://mlsanity.com/paper/1904.03181/) (2019)
- [Contextual Heterogeneous Graph Network for Human-Object Interaction Detection](https://mlsanity.com/paper/2010.10001/) (2020)
- [Cascaded Human-Object Interaction Recognition](https://mlsanity.com/paper/2003.04262/) (2020)
- [Pairwise Body-Part Attention for Recognizing Human-Object Interactions](https://mlsanity.com/paper/1807.10889/) (2018)

## Built on by (most cited first, 15 of 15)

- [GEN-VLKT: Simplify Association and Enhance Interaction Understanding for HOI Detection](https://mlsanity.com/paper/2203.13954/) (2022)
- [Detecting Any Human-Object Interaction Relationship: Universal HOI Detector with Spatial Prompt Learning on Foundation Models](https://mlsanity.com/paper/2311.03799/) (2023)
- [Exploring the Potential of Large Foundation Models for Open-Vocabulary HOI Detection](https://mlsanity.com/paper/2404.06194/) (2024)
- [Category-Aware Transformer Network for Better Human-Object Interaction Detection](https://mlsanity.com/paper/2204.04911/) (2022)
- [Re-mine, Learn and Reason: Exploring the Cross-modal Semantic Correlations for Language-guided HOI detection](https://mlsanity.com/paper/2307.13529/) (2023)
- [Towards Hard-Positive Query Mining for DETR-based Human-Object Interaction Detection](https://mlsanity.com/paper/2207.05293/) (2022)
- [HODN: Disentangling Human-Object Feature for HOI Detection](https://mlsanity.com/paper/2308.10158/) (2023)
- [Open-Vocabulary HOI Detection with Interaction-aware Prompt and Concept Calibration](https://mlsanity.com/paper/2508.03207/) (2025)
- [Exploring Self- and Cross-Triplet Correlations for Human-Object Interaction Detection](https://mlsanity.com/paper/2401.05676/) (2024)
- [CycleHOI: Improving Human-Object Interaction Detection with Cycle Consistency of Detection and Generation](https://mlsanity.com/paper/2407.11433/) (2024)
- [Towards Zero-shot Human-Object Interaction Detection via Vision-Language Integration](https://mlsanity.com/paper/2403.07246/) (2024)
- [Enhancing HOI Detection with Contextual Cues from Large Vision-Language Models](https://mlsanity.com/paper/2311.16475/) (2023)
- [Frequency-based Matcher for Long-tailed Semantic Segmentation](https://mlsanity.com/paper/2406.03917/) (2024)
- [Effective Actor-centric Human-object Interaction Detection](https://mlsanity.com/paper/2202.11998/) (2022)
- [FreeA: Human-object Interaction Detection using Free Annotation Labels](https://mlsanity.com/paper/2403.01840/) (2024)

## Abstract (arXiv)

Human-Object Interaction (HOI) detection is a fundamental task in high-level human-centric scene understanding. We propose PhraseHOI, containing a HOI branch and a novel phrase branch, to leverage language prior and improve relation expression. Specifically, the phrase branch is supervised by semantic embeddings, whose ground truths are automatically converted from the original HOI annotations without extra human efforts. Meanwhile, a novel label composition method is proposed to deal with the long-tailed problem in HOI, which composites novel phrase labels by semantic neighbors. Further, to optimize the phrase branch, a loss composed of a distilling loss and a balanced triplet loss is proposed. Extensive experiments are conducted to prove the effectiveness of the proposed PhraseHOI, which achieves significant improvement over the baseline and surpasses previous state-of-the-art methods on Full and NonRare on the challenging HICO-DET benchmark.