# Weakly-supervised HOI Detection via Prior-guided Bi-level Representation Learning

arXiv: https://arxiv.org/abs/2303.01313 · HTML: https://mlsanity.com/paper/2303.01313/

## Key facts

- Cited by 11 papers in the map (data through 14 Oct 2026).
- Builds on 36 of its 46 references that are in the map.
- First posted on arXiv 2 Mar 2023.

## Builds on (most cited first, 30 of 36)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [Microsoft COCO: Common Objects in Context](https://mlsanity.com/paper/1405.0312/) (2014)
- [Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks](https://mlsanity.com/paper/1506.01497/) (2015)
- [End-to-End Object Detection with Transformers](https://mlsanity.com/paper/2005.12872/) (2020)
- [Mask R-CNN](https://mlsanity.com/paper/1703.06870/) (2017)
- [Attention-based Deep Multiple Instance Learning](https://mlsanity.com/paper/1802.04712/) (2018)
- [Open-vocabulary Object Detection via Vision and Language Knowledge Distillation](https://mlsanity.com/paper/2104.13921/) (2021)
- [Future Frame Prediction for Anomaly Detection -- A New Baseline](https://mlsanity.com/paper/1712.09867/) (2017)
- [Learning to Prompt for Open-Vocabulary Object Detection with Vision-Language Model](https://mlsanity.com/paper/2203.14940/) (2022)
- [Learning to Detect Human-Object Interactions](https://mlsanity.com/paper/1702.05448/) (2017)
- [Visual Semantic Role Labeling](https://mlsanity.com/paper/1505.04474/) (2015)
- [HOTR: End-to-End Human-Object Interaction Detection with Transformers](https://mlsanity.com/paper/2104.13682/) (2021)
- [QPIC: Query-Based Pairwise Human-Object Interaction Detection with Image-Wide Contextual Information](https://mlsanity.com/paper/2103.05399/) (2021)
- [iCAN: Instance-Centric Attention Network for Human-Object Interaction Detection](https://mlsanity.com/paper/1808.10437/) (2018)
- [GEN-VLKT: Simplify Association and Enhance Interaction Understanding for HOI Detection](https://mlsanity.com/paper/2203.13954/) (2022)
- [DRG: Dual Relation Graph for Human-Object Interaction Detection](https://mlsanity.com/paper/2008.11714/) (2020)
- [VSGNet: Spatial Attention Network for Detecting Human Object Interactions Using Graph Convolutions](https://mlsanity.com/paper/2003.05541/) (2020)
- [Mining the Benefits of Two-stage and One-stage HOI Detection](https://mlsanity.com/paper/2108.05077/) (2021)
- [Detailed 2D-3D Joint Representation for Human-Object Interaction](https://mlsanity.com/paper/2004.08154/) (2020)
- [Spatially Conditioned Graphs for Detecting Human-Object Interactions](https://mlsanity.com/paper/2012.06060/) (2020)
- [Efficient Two-Stage Detection of Human-Object Interactions with a Novel Unary-Pairwise Transformer](https://mlsanity.com/paper/2112.01838/) (2021)
- [Pose-aware Multi-level Feature Network for Human Object Interaction Detection](https://mlsanity.com/paper/1909.08453/) (2019)
- [Self-trained Deep Ordinal Regression for End-to-End Video Anomaly Detection](https://mlsanity.com/paper/2003.06780/) (2020)
- [HOI Analysis: Integrating and Decomposing Human-Object Interaction](https://mlsanity.com/paper/2010.16219/) (2020)
- [Cascaded Human-Object Interaction Recognition](https://mlsanity.com/paper/2003.04262/) (2020)
- [Human-Object Interaction Detection via Disentangled Transformer](https://mlsanity.com/paper/2204.09290/) (2022)
- [What to look at and where: Semantic and Spatial Refined Transformer for detecting human-object interactions](https://mlsanity.com/paper/2204.00746/) (2022)
- [MSTR: Multi-Scale Transformer for End-to-End Human-Object Interaction Detection](https://mlsanity.com/paper/2203.14709/) (2022)
- [PPR-FCN: Weakly Supervised Visual Relation Detection via Parallel Pairwise R-FCN](https://mlsanity.com/paper/1708.01956/) (2017)

## Built on by (most cited first, 11 of 11)

- [HOICLIP: Efficient Knowledge Transfer for HOI Detection with Vision-Language Models](https://mlsanity.com/paper/2303.15786/) (2023)
- [Open-Vocabulary HOI Detection with Interaction-aware Prompt and Concept Calibration](https://mlsanity.com/paper/2508.03207/) (2025)
- [Exploring Self- and Cross-Triplet Correlations for Human-Object Interaction Detection](https://mlsanity.com/paper/2401.05676/) (2024)
- [Exploiting CLIP for Zero-shot HOI Detection Requires Knowledge Distillation at Multiple Levels](https://mlsanity.com/paper/2309.05069/) (2023)
- [A Review of Human-Object Interaction Detection](https://mlsanity.com/paper/2408.10641/) (2024)
- [Towards Zero-shot Human-Object Interaction Detection via Vision-Language Integration](https://mlsanity.com/paper/2403.07246/) (2024)
- [Enhancing HOI Detection with Contextual Cues from Large Vision-Language Models](https://mlsanity.com/paper/2311.16475/) (2023)
- [An analysis of HOI: using a training-free method with multimodal visual foundation models when only the test set is available, without the training set](https://mlsanity.com/paper/2408.05772/) (2024)
- [Unleashing Multimodal Large Language Models for Training-free HOI Detection in the Wild](https://mlsanity.com/paper/2607.13881/) (2026)
- [FreeA: Human-object Interaction Detection using Free Annotation Labels](https://mlsanity.com/paper/2403.01840/) (2024)
- [RegFormer: Transferable Relational Grounding for Efficient Weakly-Supervised Human-Object Interaction Detection](https://mlsanity.com/paper/2604.00507/) (2026)

## Abstract (arXiv)

Human object interaction (HOI) detection plays a crucial role in human-centric scene understanding and serves as a fundamental building-block for many vision tasks. One generalizable and scalable strategy for HOI detection is to use weak supervision, learning from image-level annotations only. This is inherently challenging due to ambiguous human-object associations, large search space of detecting HOIs and highly noisy training signal. A promising strategy to address those challenges is to exploit knowledge from large-scale pretrained models (e.g., CLIP), but a direct knowledge distillation strategy~citepliao2022gen does not perform well on the weakly-supervised setting. In contrast, we develop a CLIP-guided HOI representation capable of incorporating the prior knowledge at both image level and HOI instance level, and adopt a self-taught mechanism to prune incorrect human-object associations. Experimental results on HICO-DET and V-COCO show that our method outperforms the previous works by a sizable margin, showing the efficacy of our HOI representation.