mlsanityData to 14 Oct 2026

MSTR: Multi-Scale Transformer for End-to-End Human-Object Interaction Detection

arXiv 2203.14709 ↗First posted 28 Mar 2022cs.CVcited by 34

Not on the map. Why?

Abstract (arXiv)

Human-Object Interaction (HOI) detection is the task of identifying a set of <human, object, interaction> triplets from an image. Recent work proposed transformer encoder-decoder architectures that successfully eliminated the need for many hand-designed components in HOI detection through end-to-end training. However, they are limited to single-scale feature resolution, providing suboptimal performance in scenes containing humans, objects and their interactions with vastly different scales and distances. To tackle this problem, we propose a Multi-Scale TRansformer (MSTR) for HOI detection powered by two novel HOI-aware deformable attention modules called Dual-Entity attention and Entity-conditioned Context attention. While existing deformable attention comes at a huge cost in HOI detection performance, our proposed attention modules of MSTR learn to effectively attend to sampling points that are essential to identify interactions. In experiments, we achieve the new state-of-the-art performance on two HOI detection benchmarks.

Key facts. Cited by 34 papers in the map (data through 14 Oct 2026). Builds on 31 of its 39 references that are in the map. First posted on arXiv 28 Mar 2022.

Builds on

31 of 39 references.

References · 31

  1. 2017Attention Is All You Needcited by 60,361
  2. 2015Deep Residual Learning for Image Recognitioncited by 52,096
  3. 2020End-to-End Object Detection with Transformerscited by 7,414
  4. 2020Deformable DETR: Deformable Transformers for End-to-End Object Detectioncited by 3,422
  5. 2017Learning to Detect Human-Object Interactionscited by 240
  6. 2017Detecting and Recognizing Human-Object Interactionscited by 199
  7. 2015Visual Semantic Role Labelingcited by 190
  8. 2021HOTR: End-to-End Human-Object Interaction Detection with Transformerscited by 183
  9. 2018Learning Human-Object Interactions by Graph Parsing Neural Networkscited by 176
  10. 2021QPIC: Query-Based Pairwise Human-Object Interaction Detection with Image-Wide Contextual Informationcited by 147
  11. 2019PPDM: Parallel Point Detection and Matching for Real-time Human-Object Interaction Detectioncited by 143
  12. 2021End-to-End Human Object Interaction Detection with HOI Transformercited by 135
19 more
  1. 2020Visual Compositional Learning for Human-Object Interaction Detectioncited by 130
  2. 2018iCAN: Instance-Centric Attention Network for Human-Object Interaction Detectioncited by 127
  3. 2020DRG: Dual Relation Graph for Human-Object Interaction Detectioncited by 117
  4. 2020VSGNet: Spatial Attention Network for Detecting Human Object Interactions Using Graph Convolutionscited by 115
  5. 2020Learning Human-Object Interaction Detection using Interaction Pointscited by 108
  6. 2021Reformulating HOI Detection as Adaptive Set Predictioncited by 95
  7. 2020Detailed 2D-3D Joint Representation for Human-Object Interactioncited by 93
  8. 2019Pose-aware Multi-level Feature Network for Human Object Interaction Detectioncited by 88
  9. 2020HOI Analysis: Integrating and Decomposing Human-Object Interactioncited by 81
  10. 2020PaStaNet: Toward Human Activity Knowledge Enginecited by 80
  11. 2020Detecting Human-Object Interactions with Action Co-occurrence Priorscited by 72
  12. 2021Glance and Gaze: Inferring Action-aware Points for One-Stage Human-Object Interaction Detectioncited by 58
  13. 2019Deep Contextual Attention for Human-Object Interaction Detectioncited by 54
  14. 2019Detecting Human-Object Interactions via Functional Generalizationcited by 53
  15. 2021PnP-DETR: Towards Efficient Visual Analysis with Transformerscited by 52
  16. 2020Contextual Heterogeneous Graph Network for Human-Object Interaction Detectioncited by 50
  17. 2020ConsNet: Learning Consistency Graph for Zero-Shot Human-Object Interaction Detectioncited by 46
  18. 2021Visual Relationship Detection Using Part-and-Sum Transformers with Composite Queriescited by 31
  19. 2018Detecting Visual Relationships Using Box Attentioncited by 16

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 17; 2024: 9; 2025: 5; 2026: 3.

34 papers.

Sort:

By area:

  1. 2023Exploring Predicate Visual Context in Detecting Human-Object Interactionscited by 56
  2. 2023RLIPv2: Fast Scaling of Relational Language-Image Pre-trainingcited by 45
  3. 2023ViPLO: Vision Transformer based Pose-Conditioned Self-Loop Graph for Human-Object Interaction Detectioncited by 41
  4. 2023Relational Context Learning for Human-Object Interaction Detectioncited by 36
  5. 2023Neural-Logic Human-Object Interaction Detectioncited by 25
  6. 2024Exploring the Potential of Large Foundation Models for Open-Vocabulary HOI Detectioncited by 25
  7. 2023Efficient Multi-scale Network with Learnable Discrete Wavelet Transform for Blind Motion Deblurringcited by 24
  8. 2023Boosting Human-Object Interaction Detection with Text-to-Image Diffusion Modelcited by 22
  9. 2023Category Query Learning for Human-Object Interaction Classificationcited by 21
  10. 2024Learning from Observer Gaze:Zero-Shot Attention Prediction Oriented by Human-Object Interaction Recognitioncited by 17
  11. 2024Human-Object Interaction Detection Collaborated with Large Relation-driven Diffusion Modelscited by 15
  12. 2023Weakly-supervised HOI Detection via Prior-guided Bi-level Representation Learningcited by 11
22 more
  1. 2024Groupwise Query Specialization and Quality-Aware Multi-Assignment for Transformer-based Visual Relationship Detectioncited by 11
  2. 2023Agglomerative Transformer for Human-Object Interaction Detectioncited by 8
  3. 2025Open-Vocabulary HOI Detection with Interaction-aware Prompt and Concept Calibrationcited by 8
  4. 2023Weakly-Supervised HOI Detection from Interaction Labels Only and Language/Vision-Language Priorscited by 6
  5. 2023Exploiting CLIP for Zero-shot HOI Detection Requires Knowledge Distillation at Multiple Levelscited by 5
  6. 2024CycleHOI: Improving Human-Object Interaction Detection with Cycle Consistency of Detection and Generationcited by 5
  7. 2024A Review of Human-Object Interaction Detectioncited by 5
  8. 2025Learning Human-Object Interaction as Groupscited by 5
  9. 2023DDS: Decoupled Dynamic Scene-Graph Generation Networkcited by 3
  10. 2023Enhancing HOI Detection with Contextual Cues from Large Vision-Language Modelscited by 3
  11. 2025SGC-Net: Stratified Granular Comparison Network for Open-Vocabulary HOI Detectioncited by 3
  12. 2025Egocentric Human-Object Interaction Detection: A New Benchmark and Methodcited by 3
  13. 2025RoHOI: Robustness Benchmark for Human-Object Interaction Detectioncited by 3
  14. 2023Free-Form Composition Networks for Egocentric Action Recognitioncited by 2
  15. 2024Geometric Features Enhanced Human-Object Interaction Detectioncited by 2
  16. 2024VLM-HOI: Vision Language Models for Interpretable Human-Object Interaction Analysiscited by 1
  17. 2026Zero-shot HOI Detection with MLLM-based Detector-agnostic Interaction Recognitioncited by 1
  18. 2026ScriptHOI: Learning Scripted State Transitions for Open-Vocabulary Human-Object Interaction Detectioncited by 1
  19. 2026HOIBlender: Blending Lightweight Detection with Vision-Language Priors for Efficient Human-Object Interaction Detectioncited by 1
  20. 2023Mining Conditional Part Semantics with Occluded Extrapolation for Human-Object Interaction Detectioncited by 0
  21. 2023Disentangled Interaction Representation for One-Stage Human-Object Interaction Detectioncited by 0
  22. 2024STMixer: A One-Stage Sparse Action Detectorcited by 0