mlsanityData to 14 Oct 2026

Detecting and Recognizing Human-Object Interactions

arXiv 1704.07333 ↗First posted 24 Apr 2017cs.CVcited by 199

Not on the map. Why?

Abstract (arXiv)

To understand the visual world, a machine must not only recognize individual object instances but also how they interact. Humans are often at the center of such interactions and detecting human-object interactions is an important practical and scientific problem. In this paper, we address the task of detecting <human, verb, object> triplets in challenging everyday photos. We propose a novel model that is driven by a human-centric approach. Our hypothesis is that the appearance of a person -- their pose, clothing, action -- is a powerful cue for localizing the objects they are interacting with. To exploit this cue, our model learns to predict an action-specific density over target object locations based on the appearance of a detected person. Our model also jointly learns to detect people and objects, and by fusing these predictions it efficiently infers interaction triplets in a clean, jointly trained end-to-end system we call InteractNet. We validate our approach on the recently introduced Verbs in COCO (V-COCO) and HICO-DET datasets, where we show quantitatively compelling results.

Key facts. Cited by 199 papers in the map (data through 14 Oct 2026). First posted on arXiv 24 Apr 2017.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2021: 42; 2022: 44; 2023: 47; 2024: 28; 2025: 26; 2026: 12.

199 papers.

Sort:

By area:

  1. 2022BEVFusion: Multi-Task Multi-Sensor Fusion with Unified Bird's-Eye View Representationcited by 850
  2. 2021H2O: Two Hands Manipulating Objects for First Person Interaction Recognitioncited by 254
  3. 2021HOTR: End-to-End Human-Object Interaction Detection with Transformerscited by 183
  4. 2023InterDiff: Generating 3D Human-Object Interactions with Physics-Informed Diffusioncited by 182
  5. 2021QPIC: Query-Based Pairwise Human-Object Interaction Detection with Image-Wide Contextual Informationcited by 147
  6. 2021Towards Long-Form Video Understandingcited by 146
  7. 2021Rendezvous: Attention Mechanisms for the Recognition of Surgical Action Triplets in Endoscopic Videoscited by 141
  8. 2022EPIC-KITCHENS VISOR Benchmark: VIdeo Segmentations and Object Relationscited by 140
  9. 2021End-to-End Human Object Interaction Detection with HOI Transformercited by 135
  10. 2021Unidentified Video Objects: A Benchmark for Dense, Open-World Segmentationcited by 133
  11. 2021A Comprehensive Survey of Scene Graphs: Generation and Applicationcited by 125
  12. 2022GEN-VLKT: Simplify Association and Enhance Interaction Understanding for HOI Detectioncited by 120
38 more
  1. 2022Panoptic Scene Graph Generationcited by 114
  2. 2021Mining the Benefits of Two-stage and One-stage HOI Detectioncited by 103
  3. 2025Being-H0: Vision-Language-Action Pretraining from Large-Scale Human Videoscited by 102
  4. 2021Spatial-Temporal Transformer for Dynamic Scene Graph Generationcited by 96
  5. 2021Reformulating HOI Detection as Adaptive Set Predictioncited by 95
  6. 2021Efficient Two-Stage Detection of Human-Object Interactions with a Novel Unary-Pairwise Transformercited by 91
  7. 2022What's in your hands? 3D Reconstruction of Generic Objects in Handscited by 90
  8. 2022Compositional Human-Scene Interaction Synthesis with Semantic Controlcited by 83
  9. 2021Detecting Human-Object Interaction via Fabricated Compositional Learningcited by 79
  10. 2021Affordance Transfer Learning for Human-Object Interaction Detectioncited by 79
  11. 2022Full-Body Articulated Human-Object Interactioncited by 78
  12. 2023PhysHOI: Physics-Based Imitation of Dynamic Human-Object Interactioncited by 76
  13. 2022Scene Graph Generation: A Comprehensive Surveycited by 73
  14. 2021Video as Conditional Graph Hierarchy for Multi-Granular Question Answeringcited by 68
  15. 2021Glance and Gaze: Inferring Action-aware Points for One-Stage Human-Object Interaction Detectioncited by 58
  16. 2025LION-FS: Fast & Slow Video-Language Thinker as Online Video Assistantcited by 58
  17. 2023Exploring Predicate Visual Context in Detecting Human-Object Interactionscited by 56
  18. 2022RLIP: Relational Language-Image Pre-training for Human-Object Interaction Detectioncited by 55
  19. 2024AffordanceLLM: Grounding Affordance from Vision Language Modelscited by 55
  20. 2023Putting People in Their Place: Affordance-Aware Human Insertion into Scenescited by 53
  21. 2023LEMON: Learning 3D Human-Object Interaction Relation from 2D Imagescited by 49
  22. 2022Exploring Structure-aware Transformer over Interaction Proposals for Human-Object Interaction Detectioncited by 47
  23. 2023RLIPv2: Fast Scaling of Relational Language-Image Pre-trainingcited by 45
  24. 2022CholecTriplet2021: A benchmark challenge for surgical action triplet recognitioncited by 44
  25. 2022Interactiveness Field in Human-Object Interactionscited by 40
  26. 2022Human-Object Interaction Detection via Disentangled Transformercited by 40
  27. 2023Efficient Adaptive Human-Object Interaction Detection with Concept-guided Memorycited by 38
  28. 2022Understanding 3D Object Articulation in Internet Videoscited by 36
  29. 2022MECCANO: A Multimodal Egocentric Dataset for Humans Behavior Understanding in the Industrial-like Domaincited by 36
  30. 2024EasyHOI: Unleashing the Power of Large Models for Reconstructing Hand-Object Interactions in the Wildcited by 36
  31. 2022What to look at and where: Semantic and Spatial Refined Transformer for detecting human-object interactionscited by 35
  32. 2022MSTR: Multi-Scale Transformer for End-to-End Human-Object Interaction Detectioncited by 34
  33. 2022Mining Cross-Person Cues for Body-Part Interactiveness Learning in HOI Detectioncited by 34
  34. 2023Detecting Any Human-Object Interaction Relationship: Universal HOI Detector with Spatial Prompt Learning on Foundation Modelscited by 34
  35. 2023Understanding 3D Object Interaction from a Single Imagecited by 33
  36. 2021Visual Relationship Detection Using Part-and-Sum Transformers with Composite Queriescited by 31
  37. 2023UnionDet: Union-Level Detector Towards Real-Time Human-Object Interaction Detectioncited by 31
  38. 2022End-to-End Zero-Shot HOI Detection via Vision and Language Knowledge Distillationcited by 30

50 most cited of 199 All 199 (JSON)