mlsanityData to 7 Oct 2026

What to look at and where: Semantic and Spatial Refined Transformer for detecting human-object interactions

arXiv 2204.00746 ↗First posted 2 Apr 2022cs.CVcited by 35

Not on the map. Why?

Abstract (arXiv)

We propose a novel one-stage Transformer-based semantic and spatial refined transformer (SSRT) to solve the Human-Object Interaction detection task, which requires to localize humans and objects, and predicts their interactions. Differently from previous Transformer-based HOI approaches, which mostly focus at improving the design of the decoder outputs for the final detection, SSRT introduces two new modules to help select the most relevant object-action pairs within an image and refine the queries' representation using rich semantic and spatial features. These enhancements lead to state-of-the-art results on the two most popular HOI benchmarks: V-COCO and HICO-DET.

Key facts. Cited by 35 papers in the map (data through 7 Oct 2026). Builds on 46 of its 54 references that are in the map. First posted on arXiv 2 Apr 2022.

Builds on

46 of 54 references.

References · 46

  1. 2017Attention Is All You Needcited by 59,932
  2. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  3. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  4. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  5. 2021Swin Transformer: Hierarchical Vision Transformer using Shifted Windowscited by 12,082
  6. 2015Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networkscited by 9,617
  7. 2020End-to-End Object Detection with Transformerscited by 7,384
  8. 2021Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervisioncited by 3,848
  9. 2015Microsoft COCO Captions: Data Collection and Evaluation Servercited by 1,447
  10. 2018Unified Perceptual Parsing for Scene Understandingcited by 1,265
  11. 2017Learning to Detect Human-Object Interactionscited by 238
  12. 2016Attend, Infer, Repeat: Fast Scene Understanding with Generative Modelscited by 209
34 more
  1. 2017Detecting and Recognizing Human-Object Interactionscited by 199
  2. 2015Visual Semantic Role Labelingcited by 188
  3. 2021HOTR: End-to-End Human-Object Interaction Detection with Transformerscited by 183
  4. 2018Learning Human-Object Interactions by Graph Parsing Neural Networkscited by 176
  5. 2021QPIC: Query-Based Pairwise Human-Object Interaction Detection with Image-Wide Contextual Informationcited by 147
  6. 2019PPDM: Parallel Point Detection and Matching for Real-time Human-Object Interaction Detectioncited by 143
  7. 2021End-to-End Human Object Interaction Detection with HOI Transformercited by 134
  8. 2020Visual Compositional Learning for Human-Object Interaction Detectioncited by 129
  9. 2018iCAN: Instance-Centric Attention Network for Human-Object Interaction Detectioncited by 126
  10. 2020DRG: Dual Relation Graph for Human-Object Interaction Detectioncited by 117
  11. 2020VSGNet: Spatial Attention Network for Detecting Human Object Interactions Using Graph Convolutionscited by 115
  12. 2020Learning Human-Object Interaction Detection using Interaction Pointscited by 108
  13. 2021Reformulating HOI Detection as Adaptive Set Predictioncited by 95
  14. 2021Learning to Predict Visual Attributes in the Wildcited by 95
  15. 2020Detailed 2D-3D Joint Representation for Human-Object Interactioncited by 93
  16. 2020Spatially Conditioned Graphs for Detecting Human-Object Interactionscited by 93
  17. 2019Pose-aware Multi-level Feature Network for Human Object Interaction Detectioncited by 88
  18. 2019Learning Actor Relation Graphs for Group Activity Recognitioncited by 83
  19. 2020HOI Analysis: Integrating and Decomposing Human-Object Interactioncited by 81
  20. 2021Detecting Human-Object Interaction via Fabricated Compositional Learningcited by 79
  21. 2021Affordance Transfer Learning for Human-Object Interaction Detectioncited by 79
  22. 2020Detecting Human-Object Interactions with Action Co-occurrence Priorscited by 72
  23. 2021Glance and Gaze: Inferring Action-aware Points for One-Stage Human-Object Interaction Detectioncited by 58
  24. 2019Deep Contextual Attention for Human-Object Interaction Detectioncited by 54
  25. 2020Contextual Heterogeneous Graph Network for Human-Object Interaction Detectioncited by 50
  26. 2020ConsNet: Learning Consistency Graph for Zero-Shot Human-Object Interaction Detectioncited by 45
  27. 2018A Structured Model For Action Detectioncited by 43
  28. 2020DIRV: Dense Interaction Region Voting for End-to-End Human-Object Interaction Detectioncited by 31
  29. 2021Visual Relationship Detection Using Part-and-Sum Transformers with Composite Queriescited by 31
  30. 2021Exploiting Scene Graphs for Human-Object Interaction Detectioncited by 22
  31. 2018Actor Conditioned Attention Maps for Video Action Detectioncited by 15
  32. 2021GTNet:Guided Transformer Network for Detecting Human-Object Interactionscited by 8
  33. 2020Towards Overcoming False Positives in Visual Relationship Detectioncited by 2
  34. 2021RR-Net: Injecting Interactive Semantics in Human-Object Interaction Detectioncited by 1

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 2; 2023: 20; 2024: 10; 2025: 2; 2026: 1.

35 papers.

Sort:

By area:

  1. 2023Exploring Predicate Visual Context in Detecting Human-Object Interactionscited by 56
  2. 2023ViPLO: Vision Transformer based Pose-Conditioned Self-Loop Graph for Human-Object Interaction Detectioncited by 41
  3. 2023Efficient Adaptive Human-Object Interaction Detection with Concept-guided Memorycited by 38
  4. 2023Detecting Any Human-Object Interaction Relationship: Universal HOI Detector with Spatial Prompt Learning on Foundation Modelscited by 34
  5. 2024EZ-HOI: VLM Adaptation via Guided Prompt Learning for Zero-Shot HOI Detectioncited by 28
  6. 2023Neural-Logic Human-Object Interaction Detectioncited by 25
  7. 2024Exploring the Potential of Large Foundation Models for Open-Vocabulary HOI Detectioncited by 25
  8. 2023Category Query Learning for Human-Object Interaction Classificationcited by 21
  9. 2023Boosting Human-Object Interaction Detection with Text-to-Image Diffusion Modelcited by 21
  10. 2022CRT-6D: Fast 6D Object Pose Estimation with Cascaded Refinement Transformerscited by 15
  11. 2023Towards a Unified Transformer-based Framework for Scene Graph Generation and Human-object Interaction Detectioncited by 15
  12. 2024Human-Object Interaction Detection Collaborated with Large Relation-driven Diffusion Modelscited by 15
23 more
  1. 2024Nonverbal Interaction Detectioncited by 14
  2. 2023Re-mine, Learn and Reason: Exploring the Cross-modal Semantic Correlations for Language-guided HOI detectioncited by 13
  3. 2023MethaneMapper: Spectral Absorption aware Hyperspectral Transformer for Methane Detectioncited by 12
  4. 2023Weakly-supervised HOI Detection via Prior-guided Bi-level Representation Learningcited by 11
  5. 2024Unseen No More: Unlocking the Potential of CLIP for Generative Zero-shot HOI Detectioncited by 11
  6. 2023HODN: Disentangling Human-Object Feature for HOI Detectioncited by 9
  7. 2023Agglomerative Transformer for Human-Object Interaction Detectioncited by 8
  8. 2025Open-Vocabulary HOI Detection with Interaction-aware Prompt and Concept Calibrationcited by 7
  9. 2024CycleHOI: Improving Human-Object Interaction Detection with Cycle Consistency of Detection and Generationcited by 5
  10. 2023Focusing on what to decode and what to train: SOV Decoding with Specific Target Guided DeNoising and Vision Language Advisorcited by 4
  11. 2023Improving Human-Object Interaction Detection via Virtual Image Learningcited by 4
  12. 2022LOCL: Learning Object-Attribute Composition using Localizationcited by 3
  13. 2023DDS: Decoupled Dynamic Scene-Graph Generation Networkcited by 3
  14. 2023Human-to-Human Interaction Detectioncited by 3
  15. 2023Enhancing HOI Detection with Contextual Cues from Large Vision-Language Modelscited by 3
  16. 2024Geometric Features Enhanced Human-Object Interaction Detectioncited by 2
  17. 2025Dynamic Scene Understanding from Vision-Language Representationscited by 1
  18. 2026Unleashing Multimodal Large Language Models for Training-free HOI Detection in the Wildcited by 1
  19. 2023Mining Conditional Part Semantics with Occluded Extrapolation for Human-Object Interaction Detectioncited by 0
  20. 2023Disentangled Interaction Representation for One-Stage Human-Object Interaction Detectioncited by 0
  21. 2024STMixer: A One-Stage Sparse Action Detectorcited by 0
  22. 2024Sparse Color-Code Net: Real-Time RGB-Based 6D Object Pose Estimation on Edge Devicescited by 0
  23. 2024GrabDAE: An Innovative Framework for Unsupervised Domain Adaptation Utilizing Grab-Mask and Denoise Auto-Encodercited by 0