# Human-Object Interaction Detection via Weak Supervision

arXiv: https://arxiv.org/abs/2112.00492 · HTML: https://mlsanity.com/paper/2112.00492/

## Key facts

- Cited by 5 papers in the map (data through 14 Oct 2026).
- Builds on 27 of its 33 references that are in the map.
- First posted on arXiv 1 Dec 2021.

## Builds on (most cited first, 27 of 27)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Microsoft COCO: Common Objects in Context](https://mlsanity.com/paper/1405.0312/) (2014)
- [Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks](https://mlsanity.com/paper/1506.01497/) (2015)
- [End-to-End Object Detection with Transformers](https://mlsanity.com/paper/2005.12872/) (2020)
- [Fast R-CNN](https://mlsanity.com/paper/1504.08083/) (2015)
- [Categorical Reparameterization with Gumbel-Softmax](https://mlsanity.com/paper/1611.01144/) (2016)
- [The Concrete Distribution: A Continuous Relaxation of Discrete Random Variables](https://mlsanity.com/paper/1611.00712/) (2016)
- [R-FCN: Object Detection via Region-based Fully Convolutional Networks](https://mlsanity.com/paper/1605.06409/) (2016)
- [Learning to Detect Human-Object Interactions](https://mlsanity.com/paper/1702.05448/) (2017)
- [Detecting and Recognizing Human-Object Interactions](https://mlsanity.com/paper/1704.07333/) (2017)
- [Visual Semantic Role Labeling](https://mlsanity.com/paper/1505.04474/) (2015)
- [Convolutional Networks with Adaptive Inference Graphs](https://mlsanity.com/paper/1711.11503/) (2017)
- [QPIC: Query-Based Pairwise Human-Object Interaction Detection with Image-Wide Contextual Information](https://mlsanity.com/paper/2103.05399/) (2021)
- [PPDM: Parallel Point Detection and Matching for Real-time Human-Object Interaction Detection](https://mlsanity.com/paper/1912.12898/) (2019)
- [Visual Compositional Learning for Human-Object Interaction Detection](https://mlsanity.com/paper/2007.12407/) (2020)
- [iCAN: Instance-Centric Attention Network for Human-Object Interaction Detection](https://mlsanity.com/paper/1808.10437/) (2018)
- [DRG: Dual Relation Graph for Human-Object Interaction Detection](https://mlsanity.com/paper/2008.11714/) (2020)
- [Reformulating HOI Detection as Adaptive Set Prediction](https://mlsanity.com/paper/2103.05983/) (2021)
- [Detecting Human-Object Interactions with Action Co-occurrence Priors](https://mlsanity.com/paper/2007.08728/) (2020)
- [You Look Twice: GaterNet for Dynamic Filter Selection in CNNs](https://mlsanity.com/paper/1811.11205/) (2018)
- [Batch-Shaping for Learning Conditional Channel Gated Networks](https://mlsanity.com/paper/1907.06627/) (2019)
- [PPR-FCN: Weakly Supervised Visual Relation Detection via Parallel Pairwise R-FCN](https://mlsanity.com/paper/1708.01956/) (2017)
- [Detecting Human-Object Interaction with Mixed Supervision](https://mlsanity.com/paper/2011.04971/) (2020)
- [No-Frills Human-Object Interaction Detection: Factorization, Layout Encodings, and Training Techniques](https://mlsanity.com/paper/1811.05967/) (2018)
- [Structured Visual Search via Composition-aware Learning](https://mlsanity.com/paper/2010.14438/) (2020)
- [Self-Selective Context for Interaction Recognition](https://mlsanity.com/paper/2010.08750/) (2020)

## Built on by (most cited first, 5 of 5)

- [Weakly-supervised HOI Detection via Prior-guided Bi-level Representation Learning](https://mlsanity.com/paper/2303.01313/) (2023)
- [Exploiting CLIP for Zero-shot HOI Detection Requires Knowledge Distillation at Multiple Levels](https://mlsanity.com/paper/2309.05069/) (2023)
- [A Review of Human-Object Interaction Detection](https://mlsanity.com/paper/2408.10641/) (2024)
- [Unleashing Multimodal Large Language Models for Training-free HOI Detection in the Wild](https://mlsanity.com/paper/2607.13881/) (2026)
- [FreeA: Human-object Interaction Detection using Free Annotation Labels](https://mlsanity.com/paper/2403.01840/) (2024)

## Abstract (arXiv)

The goal of this paper is Human-object Interaction (HO-I) detection. HO-I detection aims to find interacting human-objects regions and classify their interaction from an image. Researchers obtain significant improvement in recent years by relying on strong HO-I alignment supervision from [5]. HO-I alignment supervision pairs humans with their interacted objects, and then aligns human-object pair(s) with their interaction categories. Since collecting such annotation is expensive, in this paper, we propose to detect HO-I without alignment supervision. We instead rely on image-level supervision that only enumerates existing interactions within the image without pointing where they happen. Our paper makes three contributions: i) We propose Align-Former, a visual-transformer based CNN that can detect HO-I with only image-level supervision. ii) Align-Former is equipped with HO-I align layer, that can learn to select appropriate targets to allow detector supervision. iii) We evaluate Align-Former on HICO-DET [5] and V-COCO [13], and show that Align-Former outperforms existing image-level supervised HO-I detectors by a large margin (4.71% mAP improvement from 16.14% to 20.85% on HICO-DET [5]).