mlsanityData to 13 Oct 2026

HOTR: End-to-End Human-Object Interaction Detection with Transformers

arXiv 2104.13682 ↗First posted 28 Apr 2021cs.CVcited by 183

Not on the map. Why?

Abstract (arXiv)

Human-Object Interaction (HOI) detection is a task of identifying "a set of interactions" in an image, which involves the i) localization of the subject (i.e., humans) and target (i.e., objects) of interaction, and ii) the classification of the interaction labels. Most existing methods have indirectly addressed this task by detecting human and object instances and individually inferring every pair of the detected instances. In this paper, we present a novel framework, referred to by HOTR, which directly predicts a set of <human, object, interaction> triplets from an image based on a transformer encoder-decoder architecture. Through the set prediction, our method effectively exploits the inherent semantic relationships in an image and does not require time-consuming post-processing which is the main bottleneck of existing methods. Our proposed algorithm achieves the state-of-the-art performance in two HOI detection benchmarks with an inference time under 1 ms after object detection.

Key facts. Cited by 183 papers in the map (data through 13 Oct 2026). Builds on 23 of its 33 references that are in the map. First posted on arXiv 28 Apr 2021.

Builds on

23 of 33 references.

References · 23

  1. 2015Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networkscited by 9,625
  2. 2020End-to-End Object Detection with Transformerscited by 7,396
  3. 2017Relation Networks for Object Detectioncited by 305
  4. 2017Learning to Detect Human-Object Interactionscited by 238
  5. 2017Detecting and Recognizing Human-Object Interactionscited by 199
  6. 2015Visual Semantic Role Labelingcited by 189
  7. 2018Learning Human-Object Interactions by Graph Parsing Neural Networkscited by 176
  8. 2019PPDM: Parallel Point Detection and Matching for Real-time Human-Object Interaction Detectioncited by 143
  9. 2020Visual Compositional Learning for Human-Object Interaction Detectioncited by 129
  10. 2018iCAN: Instance-Centric Attention Network for Human-Object Interaction Detectioncited by 126
  11. 2020DRG: Dual Relation Graph for Human-Object Interaction Detectioncited by 117
  12. 2020VSGNet: Spatial Attention Network for Detecting Human Object Interactions Using Graph Convolutionscited by 115
11 more
  1. 2020Learning Human-Object Interaction Detection using Interaction Pointscited by 108
  2. 2020Detailed 2D-3D Joint Representation for Human-Object Interactioncited by 93
  3. 2019Pose-aware Multi-level Feature Network for Human Object Interaction Detectioncited by 88
  4. 2020HOI Analysis: Integrating and Decomposing Human-Object Interactioncited by 81
  5. 2020PaStaNet: Toward Human Activity Knowledge Enginecited by 79
  6. 2020Detecting Human-Object Interactions with Action Co-occurrence Priorscited by 72
  7. 2019Deep Contextual Attention for Human-Object Interaction Detectioncited by 54
  8. 2019Detecting Human-Object Interactions via Functional Generalizationcited by 53
  9. 2020Contextual Heterogeneous Graph Network for Human-Object Interaction Detectioncited by 50
  10. 2020ConsNet: Learning Consistency Graph for Zero-Shot Human-Object Interaction Detectioncited by 45
  11. 2018Detecting Visual Relationships Using Box Attentioncited by 16

Built on by

Cited by year (2026 to 13 Oct)

Citing papers per year: 2021: 16; 2022: 46; 2023: 48; 2024: 36; 2025: 24; 2026: 13.

183 papers.

Sort:

By area:

  1. 2022Capturing and Inferring Dense Full-Body Human-Scene Contactcited by 177
  2. 2021SPViT: Enabling Faster Vision Transformers via Soft Token Pruningcited by 166
  3. 2021Rendezvous: Attention Mechanisms for the Recognition of Surgical Action Triplets in Endoscopic Videoscited by 139
  4. 2022GEN-VLKT: Simplify Association and Enhance Interaction Understanding for HOI Detectioncited by 119
  5. 2022Panoptic Scene Graph Generationcited by 114
  6. 2022MonoDTR: Monocular 3D Object Detection with Depth-Aware Transformercited by 108
  7. 2021Mining the Benefits of Two-stage and One-stage HOI Detectioncited by 103
  8. 2021SGTR: End-to-end Scene Graph Generation with Transformercited by 102
  9. 2021Nested Hierarchical Transformer: Towards Accurate, Data-Efficient and Interpretable Visual Understandingcited by 97
  10. 2022RelTR: Relation Transformer for Scene Graph Generationcited by 95
  11. 2021Efficient Two-Stage Detection of Human-Object Interactions with a Novel Unary-Pairwise Transformercited by 91
  12. 2022Cross-Attention of Disentangled Modalities for 3D Human Mesh Recovery with Transformerscited by 81
38 more
  1. 2023A Comprehensive Survey on Applications of Transformers for Deep Learning Taskscited by 71
  2. 2023HOICLIP: Efficient Knowledge Transfer for HOI Detection with Vision-Language Modelscited by 60
  3. 2023Exploring Predicate Visual Context in Detecting Human-Object Interactionscited by 56
  4. 2022RLIP: Relational Language-Image Pre-training for Human-Object Interaction Detectioncited by 54
  5. 2023DECO: Dense Estimation of 3D Human-Scene Contact In The Wildcited by 54
  6. 2021GiT: Graph Interactive Transformer for Vehicle Re-identificationcited by 49
  7. 2023LEMON: Learning 3D Human-Object Interaction Relation from 2D Imagescited by 49
  8. 2022Exploring Structure-aware Transformer over Interaction Proposals for Human-Object Interaction Detectioncited by 46
  9. 2022Webly Supervised Concept Expansion for General Purpose Vision Modelscited by 45
  10. 2021Structured Sparse R-CNN for Direct Scene Graph Generationcited by 44
  11. 2023RLIPv2: Fast Scaling of Relational Language-Image Pre-trainingcited by 44
  12. 2023ViPLO: Vision Transformer based Pose-Conditioned Self-Loop Graph for Human-Object Interaction Detectioncited by 41
  13. 2022Interactiveness Field in Human-Object Interactionscited by 40
  14. 2022Human-Object Interaction Detection via Disentangled Transformercited by 40
  15. 2023Efficient Adaptive Human-Object Interaction Detection with Concept-guided Memorycited by 38
  16. 2022EDTER: Edge Detection with Transformercited by 36
  17. 2022Interaction Region Visual Transformer for Egocentric Action Anticipationcited by 36
  18. 2022What to look at and where: Semantic and Spatial Refined Transformer for detecting human-object interactionscited by 35
  19. 2023Relational Context Learning for Human-Object Interaction Detectioncited by 35
  20. 2022MSTR: Multi-Scale Transformer for End-to-End Human-Object Interaction Detectioncited by 34
  21. 2022Multi-Task Learning with Multi-Query Transformer for Dense Predictioncited by 34
  22. 2022Mining Cross-Person Cues for Body-Part Interactiveness Learning in HOI Detectioncited by 34
  23. 2023Detecting Any Human-Object Interaction Relationship: Universal HOI Detector with Spatial Prompt Learning on Foundation Modelscited by 34
  24. 2022End-to-End Zero-Shot HOI Detection via Vision and Language Knowledge Distillationcited by 30
  25. 2023FGAHOI: Fine-Grained Anchors for Human-Object Interaction Detectioncited by 30
  26. 2022Rethinking Learning Approaches for Long-Term Action Anticipationcited by 29
  27. 2021QAHOI: Query-Based Anchors for Human-Object Interaction Detectioncited by 28
  28. 2022HAKE: A Knowledge Engine Foundation for Human Activity Understandingcited by 28
  29. 2024Open-World Human-Object Interaction Detection via Multi-modal Promptscited by 28
  30. 2022PSTR: End-to-End One-Step Person Search With Transformerscited by 26
  31. 2022TokenMixup: Efficient Attention-guided Token-level Data Augmentation for Transformerscited by 26
  32. 2024Exploring Conditional Multi-Modal Prompts for Zero-shot HOI Detectioncited by 26
  33. 2023Neural-Logic Human-Object Interaction Detectioncited by 25
  34. 2024Exploring the Potential of Large Foundation Models for Open-Vocabulary HOI Detectioncited by 25
  35. 2022Iterative Scene Graph Generationcited by 24
  36. 2022Peeling the Onion: Hierarchical Reduction of Data Redundancy for Efficient Vision Transformer Trainingcited by 24
  37. 2022Detecting Human-Object Interactions with Object-Guided Cross-Modal Calibrated Semanticscited by 22
  38. 2022Category-Aware Transformer Network for Better Human-Object Interaction Detectioncited by 22

50 most cited of 183 All 183 (JSON)