mlsanityData to 14 Oct 2026

Detecting Any Human-Object Interaction Relationship: Universal HOI Detector with Spatial Prompt Learning on Foundation Models

arXiv 2311.03799 ↗First posted 7 Nov 2023cs.CVcited by 34

Not on the map. Why?

Abstract (arXiv)

Human-object interaction (HOI) detection aims to comprehend the intricate relationships between humans and objects, predicting <human, action, object> triplets, and serving as the foundation for numerous computer vision tasks. The complexity and diversity of human-object interactions in the real world, however, pose significant challenges for both annotation and recognition, particularly in recognizing interactions within an open world context. This study explores the universal interaction recognition in an open-world setting through the use of Vision-Language (VL) foundation models and large language models (LLMs). The proposed method is dubbed as emphUniHOI. We conduct a deep analysis of the three hierarchical features inherent in visual HOI detectors and propose a method for high-level relation extraction aimed at VL foundation models, which we call HO prompt-based learning. Our design includes an HO Prompt-guided Decoder (HOPD), facilitates the association of high-level relation representations in the foundation model with various HO pairs within the image. Furthermore, we utilize a LLM (i.e. GPT) for interaction interpretation, generating a richer linguistic understanding for complex HOIs. For open-category interaction recognition, our method supports either of two input types: interaction phrase or interpretive sentence. Our efficient architecture design and learning methods effectively unleash the potential of the VL foundation models and LLMs, allowing UniHOI to surpass all existing methods with a substantial margin, under both supervised and zero-shot settings. The code and pre-trained weights are available at: urlhttps://github.com/Caoyichao/UniHOI.

Key facts. Cited by 34 papers in the map (data through 14 Oct 2026). Builds on 47 of its 53 references that are in the map. First posted on arXiv 7 Nov 2023.

Builds on

47 of 53 references.

References · 47

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  2. 2020End-to-End Object Detection with Transformerscited by 7,414
  3. 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,137
  4. 2022Flamingo: a Visual Language Model for Few-Shot Learningcited by 4,386
  5. 2022OPT: Open Pre-trained Transformer Language Modelscited by 3,512
  6. 2021Grounded Language-Image Pre-trainingcited by 1,156
  7. 2017Vision-and-Language Navigation: Interpreting visually-grounded navigation instructions in real environmentscited by 1,027
  8. 2021Multimodal Few-Shot Learning with Frozen Language Modelscited by 661
  9. 2021SLIP: Self-supervision meets Language-Image Pre-trainingcited by 436
  10. 2022GLIPv2: Unifying Localization and Vision-Language Understandingcited by 275
  11. 2017Learning to Detect Human-Object Interactionscited by 240
  12. 2017Detecting and Recognizing Human-Object Interactionscited by 199
35 more
  1. 2015Visual Semantic Role Labelingcited by 190
  2. 2021HOTR: End-to-End Human-Object Interaction Detection with Transformerscited by 183
  3. 2021QPIC: Query-Based Pairwise Human-Object Interaction Detection with Image-Wide Contextual Informationcited by 147
  4. 2019PPDM: Parallel Point Detection and Matching for Real-time Human-Object Interaction Detectioncited by 143
  5. 2021End-to-End Human Object Interaction Detection with HOI Transformercited by 135
  6. 2020Visual Compositional Learning for Human-Object Interaction Detectioncited by 130
  7. 2018iCAN: Instance-Centric Attention Network for Human-Object Interaction Detectioncited by 127
  8. 2022GEN-VLKT: Simplify Association and Enhance Interaction Understanding for HOI Detectioncited by 120
  9. 2020DRG: Dual Relation Graph for Human-Object Interaction Detectioncited by 117
  10. 2020VSGNet: Spatial Attention Network for Detecting Human Object Interactions Using Graph Convolutionscited by 115
  11. 2021Mining the Benefits of Two-stage and One-stage HOI Detectioncited by 103
  12. 2021Reformulating HOI Detection as Adaptive Set Predictioncited by 95
  13. 2020Detailed 2D-3D Joint Representation for Human-Object Interactioncited by 93
  14. 2020Spatially Conditioned Graphs for Detecting Human-Object Interactionscited by 93
  15. 2021Efficient Two-Stage Detection of Human-Object Interactions with a Novel Unary-Pairwise Transformercited by 91
  16. 2020HOI Analysis: Integrating and Decomposing Human-Object Interactioncited by 81
  17. 2021Detecting Human-Object Interaction via Fabricated Compositional Learningcited by 79
  18. 2021Affordance Transfer Learning for Human-Object Interaction Detectioncited by 79
  19. 2021MDMMT: Multidomain Multimodal Transformer for Video Retrievalcited by 78
  20. 2023HOICLIP: Efficient Knowledge Transfer for HOI Detection with Vision-Language Modelscited by 61
  21. 2021Glance and Gaze: Inferring Action-aware Points for One-Stage Human-Object Interaction Detectioncited by 58
  22. 2022RLIP: Relational Language-Image Pre-training for Human-Object Interaction Detectioncited by 55
  23. 2019Detecting Human-Object Interactions via Functional Generalizationcited by 53
  24. 2022Align, Reason and Learn: Enhancing Medical Vision-and-Language Pre-training with Knowledgecited by 51
  25. 2020ConsNet: Learning Consistency Graph for Zero-Shot Human-Object Interaction Detectioncited by 46
  26. 2023ViPLO: Vision Transformer based Pose-Conditioned Self-Loop Graph for Human-Object Interaction Detectioncited by 41
  27. 2022Interactiveness Field in Human-Object Interactionscited by 40
  28. 2023Relational Context Learning for Human-Object Interaction Detectioncited by 36
  29. 2022What to look at and where: Semantic and Spatial Refined Transformer for detecting human-object interactionscited by 35
  30. 2022ViSTA: Vision and Scene Text Aggregation for Cross-Modal Retrievalcited by 29
  31. 2020Polysemy Deciphering Network for Robust Human-Object Interaction Detectioncited by 25
  32. 2022Detecting Human-Object Interactions with Object-Guided Cross-Modal Calibrated Semanticscited by 22
  33. 2023Category Query Learning for Human-Object Interaction Classificationcited by 21
  34. 2021Improving Human-Object Interaction Detection via Phrase Learning and Label Compositioncited by 15
  35. 2023Re-mine, Learn and Reason: Exploring the Cross-modal Semantic Correlations for Language-guided HOI detectioncited by 13

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 1; 2024: 6; 2025: 19; 2026: 8.

34 papers.

Sort:

By area:

  1. 2024EZ-HOI: VLM Adaptation via Guided Prompt Learning for Zero-Shot HOI Detectioncited by 28
  2. 2024Exploring the Potential of Large Foundation Models for Open-Vocabulary HOI Detectioncited by 25
  3. 2025Object Detection with Multimodal Large Vision-Language Models: An In-depth Reviewcited by 14
  4. 2025HOLa: Zero-Shot HOI Detection with Low-Rank Decomposed VLM Feature Adaptationcited by 12
  5. 2024Unseen No More: Unlocking the Potential of CLIP for Generative Zero-shot HOI Detectioncited by 11
  6. 2025Open-Vocabulary HOI Detection with Interaction-aware Prompt and Concept Calibrationcited by 8
  7. 2025Learning Human-Object Interaction as Groupscited by 5
  8. 2024UAHOI: Uncertainty-aware Robust Interaction Learning for HOI Detectioncited by 4
  9. 2023Enhancing HOI Detection with Contextual Cues from Large Vision-Language Modelscited by 3
  10. 2025SGC-Net: Stratified Granular Comparison Network for Open-Vocabulary HOI Detectioncited by 3
  11. 2025RoHOI: Robustness Benchmark for Human-Object Interaction Detectioncited by 3
  12. 2025No More Sibling Rivalry: Debiasing Human-Object Interaction Detectioncited by 3
22 more
  1. 2025Visual Diversity and Region-aware Prompt Learning for Zero-shot HOI Detectioncited by 3
  2. 2025Decoupled Generative Modeling for Human-Object Interaction Synthesiscited by 3
  3. 2024Orchestrating the Symphony of Prompt Distribution Learning for Human-Object Interaction Detectioncited by 2
  4. 2025Dynamic Scoring with Enhanced Semantics for Training-Free Human-Object Interaction Detectioncited by 2
  5. 2026Streamlined Open-Vocabulary Human-Object Interaction Detectioncited by 2
  6. 2024CL-HOI: Cross-Level Human-Object Interaction Distillation from Vision Large Language Modelscited by 1
  7. 2025HOI-R1: Exploring the Potential of Multimodal Large Language Models for Human-Object Interaction Detectioncited by 1
  8. 2025DecoDINO: 3D Human-Scene Contact Prediction with Semantic Classificationcited by 1
  9. 2025Mitigating Long-Tail Bias in HOI Detection via Adaptive Diversity Cachecited by 1
  10. 2026Zero-shot HOI Detection with MLLM-based Detector-agnostic Interaction Recognitioncited by 1
  11. 2026SHOE: Semantic HOI Open-Vocabulary Evaluation Metriccited by 1
  12. 2026Mining Instance-Centric Vision-Language Contexts for Human-Object Interaction Detectioncited by 1
  13. 2026Unleashing Multimodal Large Language Models for Training-free HOI Detection in the Wildcited by 1
  14. 2026HOIBlender: Blending Lightweight Detection with Vision-Language Priors for Efficient Human-Object Interaction Detectioncited by 1
  15. 2025B-RIGHT: Benchmark Re-evaluation for Integrity in Generalized Human-Object Interaction Testingcited by 0
  16. 2025Separated Inter/Intra-Modal Fusion Prompts for Compositional Zero-Shot Learningcited by 0
  17. 2025On the Robustness of Human-Object Interaction Detection against Distribution Shiftcited by 0
  18. 2025From Semantics, Scene to Instance-awareness: Distilling Foundation Model for Grounded Open-vocabulary Situation Recognitioncited by 0
  19. 2025Part-Aware Bottom-Up Group Reasoning for Fine-Grained Social Interaction Detectioncited by 0
  20. 2025Generative Human-Object Interaction Detection via Differentiable Cognitive Steering of Multi-modal LLMscited by 0
  21. 2026OneHOI: Unifying Human-Object Interaction Generation and Editingcited by 0
  22. 2026ByteAction: Byte-space Action Recognition Foundation Modelcited by 0