mlsanityData to 14 Oct 2026

VLM-HOI: Vision Language Models for Interpretable Human-Object Interaction Analysis

arXiv 2411.18038 ↗First posted 27 Nov 2024cs.CV cs.AIcited by 1

On the map: Multimodal LLMs / vision-language models → Driving VLMs and open-vocabulary detection

Abstract (arXiv)

The Large Vision Language Model (VLM) has recently addressed remarkable progress in bridging two fundamental modalities. VLM, trained by a sufficiently large dataset, exhibits a comprehensive understanding of both visual and linguistic to perform diverse tasks. To distill this knowledge accurately, in this paper, we introduce a novel approach that explicitly utilizes VLM as an objective function form for the Human-Object Interaction (HOI) detection task (VLM-HOI). Specifically, we propose a method that quantifies the similarity of the predicted HOI triplet using the Image-Text matching technique. We represent HOI triplets linguistically to fully utilize the language comprehension of VLMs, which are more suitable than CLIP models due to their localization and object-centric nature. This matching score is used as an objective for contrastive optimization. To our knowledge, this is the first utilization of VLM language abilities for HOI detection. Experiments demonstrate the effectiveness of our method, achieving state-of-the-art HOI detection accuracy on benchmarks. We believe integrating VLMs into HOI detection represents important progress towards more advanced and interpretable analysis of human-object interactions.

Key facts. Cited by 1 paper in the map (data through 14 Oct 2026). Placed in Driving VLMs and open-vocabulary detection, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 53 of its 56 references that are in the map. First posted on arXiv 27 Nov 2024.

Builds on

53 of 56 references.

Closest ancestors · 1

Same direction, newest first

  1. 2023HOICLIP: Efficient Knowledge Transfer for HOI Detection with Vision-Language Models28 Mar

Other references · 52

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  2. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  3. 2020Language Models are Few-Shot Learnerscited by 32,419
  4. 2014Microsoft COCO: Common Objects in Contextcited by 17,608
  5. 2023Visual Instruction Tuningcited by 7,869
  6. 2020End-to-End Object Detection with Transformerscited by 7,414
  7. 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,137
  8. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,326
  9. 2022BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generationcited by 4,322
  10. 2023InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuningcited by 2,876
  11. 2015VQA: Visual Question Answeringcited by 2,506
  12. 2021Align before Fuse: Vision and Language Representation Learning with Momentum Distillationcited by 1,619
38 more
  1. 2019LXMERT: Learning Cross-Modality Encoder Representations from Transformerscited by 1,553
  2. 2015Microsoft COCO Captions: Data Collection and Evaluation Servercited by 1,462
  3. 2015Show, Attend and Tell: Neural Image Caption Generation with Visual Attentioncited by 1,423
  4. 2017Bottom-Up and Top-Down Attention for Image Captioning and Visual Question Answeringcited by 1,312
  5. 2019VisualBERT: A Simple and Performant Baseline for Vision and Languagecited by 1,296
  6. 2022CoCa: Contrastive Captioners are Image-Text Foundation Modelscited by 1,274
  7. 2018Unified Perceptual Parsing for Scene Understandingcited by 1,271
  8. 2018Spatial Temporal Graph Convolutional Networks for Skeleton-Based Action Recognitioncited by 1,209
  9. 2017Multimodal Machine Learning: A Survey and Taxonomycited by 1,013
  10. 2014Deep Visual-Semantic Alignments for Generating Image Descriptionscited by 968
  11. 2019VL-BERT: Pre-training of Generic Visual-Linguistic Representationscited by 954
  12. 2014Show and Tell: A Neural Image Caption Generatorcited by 879
  13. 2019Unified Vision-Language Pre-Training for Image Captioning and VQAcited by 494
  14. 201912-in-1: Multi-Task Vision and Language Representation Learningcited by 278
  15. 2018Exploring Visual Relationship for Image Captioningcited by 215
  16. 2014Unifying Visual-Semantic Embeddings with Multimodal Neural Language Modelscited by 198
  17. 2015Visual Semantic Role Labelingcited by 190
  18. 2021HOTR: End-to-End Human-Object Interaction Detection with Transformerscited by 183
  19. 2021QPIC: Query-Based Pairwise Human-Object Interaction Detection with Image-Wide Contextual Informationcited by 147
  20. 2019PPDM: Parallel Point Detection and Matching for Real-time Human-Object Interaction Detectioncited by 143
  21. 2021End-to-End Human Object Interaction Detection with HOI Transformercited by 135
  22. 2020Visual Compositional Learning for Human-Object Interaction Detectioncited by 130
  23. 2018iCAN: Instance-Centric Attention Network for Human-Object Interaction Detectioncited by 127
  24. 2022GEN-VLKT: Simplify Association and Enhance Interaction Understanding for HOI Detectioncited by 120
  25. 2021Mining the Benefits of Two-stage and One-stage HOI Detectioncited by 103
  26. 2020Spatially Conditioned Graphs for Detecting Human-Object Interactionscited by 93
  27. 2021Efficient Two-Stage Detection of Human-Object Interactions with a Novel Unary-Pairwise Transformercited by 91
  28. 2020HOI Analysis: Integrating and Decomposing Human-Object Interactioncited by 81
  29. 2021Glance and Gaze: Inferring Action-aware Points for One-Stage Human-Object Interaction Detectioncited by 58
  30. 2023Visibility Aware Human-Object Interaction Tracking from Single RGB Cameracited by 58
  31. 2023Exploring Predicate Visual Context in Detecting Human-Object Interactionscited by 56
  32. 2022RLIP: Relational Language-Image Pre-training for Human-Object Interaction Detectioncited by 55
  33. 2022Exploring Structure-aware Transformer over Interaction Proposals for Human-Object Interaction Detectioncited by 47
  34. 2023RLIPv2: Fast Scaling of Relational Language-Image Pre-trainingcited by 45
  35. 2023ViPLO: Vision Transformer based Pose-Conditioned Self-Loop Graph for Human-Object Interaction Detectioncited by 41
  36. 2022Human-Object Interaction Detection via Disentangled Transformercited by 40
  37. 2023Relational Context Learning for Human-Object Interaction Detectioncited by 36
  38. 2022MSTR: Multi-Scale Transformer for End-to-End Human-Object Interaction Detectioncited by 34

50 most cited of 52 All 52 (JSON)

Built on by

Cited by year

Citing papers per year: 2025: 1.

1 paper.

  1. 2025RoHOI: Robustness Benchmark for Human-Object Interaction Detectioncited by 3

Similar, not cited

Close in wording, no citation either way

  1. 2026Towards Unconstrained Human-Object Interaction
  2. 2023Enhancing HOI Detection with Contextual Cues from Large Vision-Language Models
  3. 2024CL-HOI: Cross-Level Human-Object Interaction Distillation from Vision Large Language Models
  4. 2025HOI-R1: Exploring the Potential of Multimodal Large Language Models for Human-Object Interaction Detection
  5. 2025Bilateral Collaboration with Large Vision-Language Models for Open Vocabulary Human-Object Interaction Detection
  6. 2025CrossHOI-Bench: A Unified Benchmark for HOI Evaluation across Vision-Language Models and HOI-Specific Methods
  7. 2026Streamlined Open-Vocabulary Human-Object Interaction Detection
  8. 2026Harnessing Multimodal Large Language Models for Training-Free Human-Object Interaction Detection