# Compositional Learning in Transformer-Based Human-Object Interaction Detection

arXiv: https://arxiv.org/abs/2308.05961 · HTML: https://mlsanity.com/paper/2308.05961/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 20 of its 22 references that are in the map.
- First posted on arXiv 11 Aug 2023.

## Builds on (most cited first, 20 of 20)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [Microsoft COCO: Common Objects in Context](https://mlsanity.com/paper/1405.0312/) (2014)
- [Deformable DETR: Deformable Transformers for End-to-End Object Detection](https://mlsanity.com/paper/2010.04159/) (2020)
- [Generalized Intersection over Union: A Metric and A Loss for Bounding Box Regression](https://mlsanity.com/paper/1902.09630/) (2019)
- [Learning to Detect Human-Object Interactions](https://mlsanity.com/paper/1702.05448/) (2017)
- [Visual Semantic Role Labeling](https://mlsanity.com/paper/1505.04474/) (2015)
- [HOTR: End-to-End Human-Object Interaction Detection with Transformers](https://mlsanity.com/paper/2104.13682/) (2021)
- [QPIC: Query-Based Pairwise Human-Object Interaction Detection with Image-Wide Contextual Information](https://mlsanity.com/paper/2103.05399/) (2021)
- [PPDM: Parallel Point Detection and Matching for Real-time Human-Object Interaction Detection](https://mlsanity.com/paper/1912.12898/) (2019)
- [Visual Compositional Learning for Human-Object Interaction Detection](https://mlsanity.com/paper/2007.12407/) (2020)
- [GEN-VLKT: Simplify Association and Enhance Interaction Understanding for HOI Detection](https://mlsanity.com/paper/2203.13954/) (2022)
- [Mining the Benefits of Two-stage and One-stage HOI Detection](https://mlsanity.com/paper/2108.05077/) (2021)
- [Efficient Two-Stage Detection of Human-Object Interactions with a Novel Unary-Pairwise Transformer](https://mlsanity.com/paper/2112.01838/) (2021)
- [Pose-aware Multi-level Feature Network for Human Object Interaction Detection](https://mlsanity.com/paper/1909.08453/) (2019)
- [Detecting Human-Object Interaction via Fabricated Compositional Learning](https://mlsanity.com/paper/2103.08214/) (2021)
- [Affordance Transfer Learning for Human-Object Interaction Detection](https://mlsanity.com/paper/2104.02867/) (2021)
- [Detecting Human-Object Interactions via Functional Generalization](https://mlsanity.com/paper/1904.03181/) (2019)
- [ConsNet: Learning Consistency Graph for Zero-Shot Human-Object Interaction Detection](https://mlsanity.com/paper/2008.06254/) (2020)
- [QAHOI: Query-Based Anchors for Human-Object Interaction Detection](https://mlsanity.com/paper/2112.08647/) (2021)

## Built on by (most cited first, 2 of 2)

- [On the Robustness of Human-Object Interaction Detection against Distribution Shift](https://mlsanity.com/paper/2506.18021/) (2025)
- [Funnel-HOI: Top-Down Perception for Zero-Shot HOI Detection](https://mlsanity.com/paper/2507.12628/) (2025)

## Abstract (arXiv)

Human-object interaction (HOI) detection is an important part of understanding human activities and visual scenes. The long-tailed distribution of labeled instances is a primary challenge in HOI detection, promoting research in few-shot and zero-shot learning. Inspired by the combinatorial nature of HOI triplets, some existing approaches adopt the idea of compositional learning, in which object and action features are learned individually and re-composed as new training samples. However, these methods follow the CNN-based two-stage paradigm with limited feature extraction ability, and often rely on auxiliary information for better performance. Without introducing any additional information, we creatively propose a transformer-based framework for compositional HOI learning. Human-object pair representations and interaction representations are re-composed across different HOI instances, which involves richer contextual information and promotes the generalization of knowledge. Experiments show our simple but effective method achieves state-of-the-art performance, especially on rare HOI classes.