# Detecting and Recognizing Human-Object Interactions

arXiv: https://arxiv.org/abs/1704.07333 · HTML: https://mlsanity.com/paper/1704.07333/

## Key facts

- Cited by 199 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 24 Apr 2017.

## Built on by (most cited first, 30 of 199)

- [BEVFusion: Multi-Task Multi-Sensor Fusion with Unified Bird's-Eye View Representation](https://mlsanity.com/paper/2205.13542/) (2022)
- [H2O: Two Hands Manipulating Objects for First Person Interaction Recognition](https://mlsanity.com/paper/2104.11181/) (2021)
- [HOTR: End-to-End Human-Object Interaction Detection with Transformers](https://mlsanity.com/paper/2104.13682/) (2021)
- [InterDiff: Generating 3D Human-Object Interactions with Physics-Informed Diffusion](https://mlsanity.com/paper/2308.16905/) (2023)
- [QPIC: Query-Based Pairwise Human-Object Interaction Detection with Image-Wide Contextual Information](https://mlsanity.com/paper/2103.05399/) (2021)
- [Towards Long-Form Video Understanding](https://mlsanity.com/paper/2106.11310/) (2021)
- [Rendezvous: Attention Mechanisms for the Recognition of Surgical Action Triplets in Endoscopic Videos](https://mlsanity.com/paper/2109.03223/) (2021)
- [EPIC-KITCHENS VISOR Benchmark: VIdeo Segmentations and Object Relations](https://mlsanity.com/paper/2209.13064/) (2022)
- [End-to-End Human Object Interaction Detection with HOI Transformer](https://mlsanity.com/paper/2103.04503/) (2021)
- [Unidentified Video Objects: A Benchmark for Dense, Open-World Segmentation](https://mlsanity.com/paper/2104.04691/) (2021)
- [A Comprehensive Survey of Scene Graphs: Generation and Application](https://mlsanity.com/paper/2104.01111/) (2021)
- [GEN-VLKT: Simplify Association and Enhance Interaction Understanding for HOI Detection](https://mlsanity.com/paper/2203.13954/) (2022)
- [Panoptic Scene Graph Generation](https://mlsanity.com/paper/2207.11247/) (2022)
- [Mining the Benefits of Two-stage and One-stage HOI Detection](https://mlsanity.com/paper/2108.05077/) (2021)
- [Being-H0: Vision-Language-Action Pretraining from Large-Scale Human Videos](https://mlsanity.com/paper/2507.15597/) (2025)
- [Spatial-Temporal Transformer for Dynamic Scene Graph Generation](https://mlsanity.com/paper/2107.12309/) (2021)
- [Reformulating HOI Detection as Adaptive Set Prediction](https://mlsanity.com/paper/2103.05983/) (2021)
- [Efficient Two-Stage Detection of Human-Object Interactions with a Novel Unary-Pairwise Transformer](https://mlsanity.com/paper/2112.01838/) (2021)
- [What's in your hands? 3D Reconstruction of Generic Objects in Hands](https://mlsanity.com/paper/2204.07153/) (2022)
- [Compositional Human-Scene Interaction Synthesis with Semantic Control](https://mlsanity.com/paper/2207.12824/) (2022)
- [Detecting Human-Object Interaction via Fabricated Compositional Learning](https://mlsanity.com/paper/2103.08214/) (2021)
- [Affordance Transfer Learning for Human-Object Interaction Detection](https://mlsanity.com/paper/2104.02867/) (2021)
- [Full-Body Articulated Human-Object Interaction](https://mlsanity.com/paper/2212.10621/) (2022)
- [PhysHOI: Physics-Based Imitation of Dynamic Human-Object Interaction](https://mlsanity.com/paper/2312.04393/) (2023)
- [Scene Graph Generation: A Comprehensive Survey](https://mlsanity.com/paper/2201.00443/) (2022)
- [Video as Conditional Graph Hierarchy for Multi-Granular Question Answering](https://mlsanity.com/paper/2112.06197/) (2021)
- [Glance and Gaze: Inferring Action-aware Points for One-Stage Human-Object Interaction Detection](https://mlsanity.com/paper/2104.05269/) (2021)
- [LION-FS: Fast & Slow Video-Language Thinker as Online Video Assistant](https://mlsanity.com/paper/2503.03663/) (2025)
- [Exploring Predicate Visual Context in Detecting Human-Object Interactions](https://mlsanity.com/paper/2308.06202/) (2023)
- [RLIP: Relational Language-Image Pre-training for Human-Object Interaction Detection](https://mlsanity.com/paper/2209.01814/) (2022)

## Abstract (arXiv)

To understand the visual world, a machine must not only recognize individual object instances but also how they interact. Humans are often at the center of such interactions and detecting human-object interactions is an important practical and scientific problem. In this paper, we address the task of detecting <human, verb, object> triplets in challenging everyday photos. We propose a novel model that is driven by a human-centric approach. Our hypothesis is that the appearance of a person -- their pose, clothing, action -- is a powerful cue for localizing the objects they are interacting with. To exploit this cue, our model learns to predict an action-specific density over target object locations based on the appearance of a detected person. Our model also jointly learns to detect people and objects, and by fusing these predictions it efficiently infers interaction triplets in a clean, jointly trained end-to-end system we call InteractNet. We validate our approach on the recently introduced Verbs in COCO (V-COCO) and HICO-DET datasets, where we show quantitatively compelling results.