mlsanityData to 14 Oct 2026

Exploring Interactive Semantic Alignment for Efficient HOI Detection with Vision-language Model

arXiv 2404.12678 ↗First posted 19 Apr 2024cs.CVcited by 0

On the map: Multimodal LLMs / vision-language models → Driving VLMs and open-vocabulary detection

Abstract (arXiv)

Human-Object Interaction (HOI) detection aims to localize human-object pairs and comprehend their interactions. Recently, two-stage transformer-based methods have demonstrated competitive performance. However, these methods frequently focus on object appearance features and ignore global contextual information. Besides, vision-language model CLIP which effectively aligns visual and text embeddings has shown great potential in zero-shot HOI detection. Based on the former facts, We introduce a novel HOI detector named ISA-HOI, which extensively leverages knowledge from CLIP, aligning interactive semantics between visual and textual features. We first extract global context of image and local features of object to Improve interaction Features in images (IF). On the other hand, we propose a Verb Semantic Improvement (VSI) module to enhance textual features of verb labels via cross-modal fusion. Ultimately, our method achieves competitive results on the HICO-DET and V-COCO benchmarks with much fewer training epochs, and outperforms the state-of-the-art under zero-shot settings.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in Driving VLMs and open-vocabulary detection, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 25 of its 26 references that are in the map. First posted on arXiv 19 Apr 2024.

Builds on

25 of 26 references.

Closest ancestors · 1

Same direction, newest first

  1. 2023HOICLIP: Efficient Knowledge Transfer for HOI Detection with Vision-Language Models28 Mar

Other references · 24

  1. 2017Attention Is All You Needcited by 60,361
  2. 2015Deep Residual Learning for Image Recognitioncited by 52,096
  3. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  4. 2021Swin Transformer: Hierarchical Vision Transformer using Shifted Windowscited by 12,139
  5. 2020End-to-End Object Detection with Transformerscited by 7,414
  6. 2017Mask R-CNNcited by 7,277
  7. 2016Layer Normalizationcited by 5,669
  8. 2017Learning to Detect Human-Object Interactionscited by 240
  9. 2015Visual Semantic Role Labelingcited by 190
  10. 2022ZegCLIP: Towards Adapting CLIP for Zero-shot Semantic Segmentationcited by 178
  11. 2022DETRs with Hybrid Matchingcited by 150
  12. 2021QPIC: Query-Based Pairwise Human-Object Interaction Detection with Image-Wide Contextual Informationcited by 147
12 more
  1. 2022GEN-VLKT: Simplify Association and Enhance Interaction Understanding for HOI Detectioncited by 120
  2. 2021Mining the Benefits of Two-stage and One-stage HOI Detectioncited by 103
  3. 2020Spatially Conditioned Graphs for Detecting Human-Object Interactionscited by 93
  4. 2021Efficient Two-Stage Detection of Human-Object Interactions with a Novel Unary-Pairwise Transformercited by 91
  5. 2021Detecting Human-Object Interaction via Fabricated Compositional Learningcited by 79
  6. 2023Exploring Predicate Visual Context in Detecting Human-Object Interactionscited by 56
  7. 2022RLIP: Relational Language-Image Pre-training for Human-Object Interaction Detectioncited by 55
  8. 2022Exploring Structure-aware Transformer over Interaction Proposals for Human-Object Interaction Detectioncited by 47
  9. 2022Human-Object Interaction Detection via Disentangled Transformercited by 40
  10. 2023Efficient Adaptive Human-Object Interaction Detection with Concept-guided Memorycited by 38
  11. 2022End-to-End Zero-Shot HOI Detection via Vision and Language Knowledge Distillationcited by 30
  12. 2022Detecting Human-Object Interactions with Object-Guided Cross-Modal Calibrated Semanticscited by 22

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2024Towards Zero-shot Human-Object Interaction Detection via Vision-Language Integration
  2. 2024Exploring the Potential of Large Foundation Models for Open-Vocabulary HOI Detection
  3. 2026Streamlined Open-Vocabulary Human-Object Interaction Detection
  4. 2023Exploiting CLIP for Zero-shot HOI Detection Requires Knowledge Distillation at Multiple Levels
  5. 2025Bilateral Collaboration with Large Vision-Language Models for Open Vocabulary Human-Object Interaction Detection
  6. 2024VLM-HOI: Vision Language Models for Interpretable Human-Object Interaction Analysis
  7. 2026HOIBlender: Blending Lightweight Detection with Vision-Language Priors for Efficient Human-Object Interaction Detection
  8. 2026Towards Unconstrained Human-Object Interaction