Data to 7 Oct 2026

End-to-End Human Object Interaction Detection with HOI Transformer

arXiv 2103.04503 ↗First posted 8 Mar 2021cs.CVcited by 134

Not on the map. Why?

Abstract (arXiv)

We propose HOI Transformer to tackle human object interaction (HOI) detection in an end-to-end manner. Current approaches either decouple HOI task into separated stages of object detection and interaction classification or introduce surrogate interaction problem. In contrast, our method, named HOI Transformer, streamlines the HOI pipeline by eliminating the need for many hand-designed components. HOI Transformer reasons about the relations of objects and humans from global image context and directly predicts HOI instances in parallel. A quintuple matching loss is introduced to force HOI predictions in a unified way. Our method is conceptually much simpler and demonstrates improved accuracy. Without bells and whistles, HOI Transformer achieves 26.61% AP on HICO-DET and 52.9% AProle on V-COCO, surpassing previous methods with the advantage of being much simpler. We hope our approach will serve as a simple and effective alternative for HOI tasks. Code is available at https://github.com/bbepoch/HoiTransformer .

Key facts. Cited by 134 papers in the map (data through 7 Oct 2026). Builds on 27 of its 40 references that are in the map. First posted on arXiv 8 Mar 2021.

Builds on

27 of 40 references.

References · 27

2017Attention Is All You Needcited by 59,932
2015Deep Residual Learning for Image Recognitioncited by 51,809
2014Microsoft COCO: Common Objects in Contextcited by 17,498
2020End-to-End Object Detection with Transformerscited by 7,384
2019Generalized Intersection over Union: A Metric and A Loss for Bounding Box Regressioncited by 1,361
2019Objects as Pointscited by 1,172
2018Image Transformercited by 770
2017Non-Autoregressive Neural Machine Translationcited by 457
2019Attention Augmented Convolutional Networkscited by 363
2017Learning to Detect Human-Object Interactionscited by 238
2017Parallel WaveNet: Fast High-Fidelity Speech Synthesiscited by 214
2017Detecting and Recognizing Human-Object Interactionscited by 199

15 more

2015Visual Semantic Role Labelingcited by 188
2018Learning Human-Object Interactions by Graph Parsing Neural Networkscited by 176
2019PPDM: Parallel Point Detection and Matching for Real-time Human-Object Interaction Detectioncited by 143
2015End-to-end people detection in crowded scenescited by 140
2020Visual Compositional Learning for Human-Object Interaction Detectioncited by 129
2018iCAN: Instance-Centric Attention Network for Human-Object Interaction Detectioncited by 126
2020DRG: Dual Relation Graph for Human-Object Interaction Detectioncited by 117
2020VSGNet: Spatial Attention Network for Detecting Human Object Interactions Using Graph Convolutionscited by 115
2020Learning Human-Object Interaction Detection using Interaction Pointscited by 108
2019Pose-aware Multi-level Feature Network for Human Object Interaction Detectioncited by 88
2020PaStaNet: Toward Human Activity Knowledge Enginecited by 79
2020Detecting Human-Object Interactions with Action Co-occurrence Priorscited by 72
2019Detecting Human-Object Interactions via Functional Generalizationcited by 53
2020Contextual Heterogeneous Graph Network for Human-Object Interaction Detectioncited by 50
2020Cascaded Human-Object Interaction Recognitioncited by 40

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 17; 2022: 39; 2023: 35; 2024: 21; 2025: 17; 2026: 5.

134 papers.

Sort:

By area:

2022Test-Time Prompt Tuning for Zero-Shot Generalization in Vision-Language Modelscited by 379
2022Capturing and Inferring Dense Full-Body Human-Scene Contactcited by 176
2021Rendezvous: Attention Mechanisms for the Recognition of Surgical Action Triplets in Endoscopic Videoscited by 139
2022GEN-VLKT: Simplify Association and Enhance Interaction Understanding for HOI Detectioncited by 119
2022Panoptic Scene Graph Generationcited by 114
2022MonoDTR: Monocular 3D Object Detection with Depth-Aware Transformercited by 108
2021Mining the Benefits of Two-stage and One-stage HOI Detectioncited by 103
2021SGTR: End-to-end Scene Graph Generation with Transformercited by 102
2022RelTR: Relation Transformer for Scene Graph Generationcited by 95
2021Efficient Two-Stage Detection of Human-Object Interactions with a Novel Unary-Pairwise Transformercited by 91
2021KVT: k-NN Attention for Boosting Vision Transformerscited by 64
2023HOICLIP: Efficient Knowledge Transfer for HOI Detection with Vision-Language Modelscited by 60

38 more

2022Human-Aware Object Placement for Visual Environment Reconstructioncited by 56
2023Exploring Predicate Visual Context in Detecting Human-Object Interactionscited by 56
2022RLIP: Relational Language-Image Pre-training for Human-Object Interaction Detectioncited by 54
2023DECO: Dense Estimation of 3D Human-Scene Contact In The Wildcited by 54
2023LEMON: Learning 3D Human-Object Interaction Relation from 2D Imagescited by 49
2022Exploring Structure-aware Transformer over Interaction Proposals for Human-Object Interaction Detectioncited by 46
2022Webly Supervised Concept Expansion for General Purpose Vision Modelscited by 45
2021Structured Sparse R-CNN for Direct Scene Graph Generationcited by 44
2023RLIPv2: Fast Scaling of Relational Language-Image Pre-trainingcited by 44
2023ViPLO: Vision Transformer based Pose-Conditioned Self-Loop Graph for Human-Object Interaction Detectioncited by 41
2024AiOS: All-in-One-Stage Expressive Human Pose and Shape Estimationcited by 41
2022Interactiveness Field in Human-Object Interactionscited by 40
2022Human-Object Interaction Detection via Disentangled Transformercited by 40
2023Efficient Adaptive Human-Object Interaction Detection with Concept-guided Memorycited by 38
2022EDTER: Edge Detection with Transformercited by 35
2022What to look at and where: Semantic and Spatial Refined Transformer for detecting human-object interactionscited by 35
2023Relational Context Learning for Human-Object Interaction Detectioncited by 35
2022MSTR: Multi-Scale Transformer for End-to-End Human-Object Interaction Detectioncited by 34
2022Bongard-HOI: Benchmarking Few-Shot Visual Reasoning for Human-Object Interactionscited by 34
2022Mining Cross-Person Cues for Body-Part Interactiveness Learning in HOI Detectioncited by 34
2023Detecting Any Human-Object Interaction Relationship: Universal HOI Detector with Spatial Prompt Learning on Foundation Modelscited by 34
2021Visual Relationship Detection Using Part-and-Sum Transformers with Composite Queriescited by 31
2024Just Shift It: Test-Time Prototype Shifting for Zero-Shot Generalization with Vision-Language Modelscited by 31
2022End-to-End Zero-Shot HOI Detection via Vision and Language Knowledge Distillationcited by 30
2023FGAHOI: Fine-Grained Anchors for Human-Object Interaction Detectioncited by 30
2022Rethinking Learning Approaches for Long-Term Action Anticipationcited by 29
2021QAHOI: Query-Based Anchors for Human-Object Interaction Detectioncited by 28
2024EZ-HOI: VLM Adaptation via Guided Prompt Learning for Zero-Shot HOI Detectioncited by 28
2024Exploring Conditional Multi-Modal Prompts for Zero-shot HOI Detectioncited by 26
2023Neural-Logic Human-Object Interaction Detectioncited by 25
2021Classification-Then-Grounding: Reformulating Video Scene Graphs as Temporal Bipartite Graphscited by 23
2022Detecting Human-Object Interactions with Object-Guided Cross-Modal Calibrated Semanticscited by 22
2022Category-Aware Transformer Network for Better Human-Object Interaction Detectioncited by 22
2023CholecTriplet2022: Show me a tool and tell me the triplet -- an endoscopic vision challenge for surgical action triplet detectioncited by 22
2023Category Query Learning for Human-Object Interaction Classificationcited by 21
2022Detector-Free Weakly Supervised Group Activity Recognitioncited by 20
2022Iwin: Human-Object Interaction Detection via Transformer with Irregular Windowscited by 19
2024Learning from Observer Gaze:Zero-Shot Attention Prediction Oriented by Human-Object Interaction Recognitioncited by 17

50 most cited of 134 All 134 (JSON)