mlsanityData to 7 Oct 2026

HOICLIP: Efficient Knowledge Transfer for HOI Detection with Vision-Language Models

arXiv 2303.15786 ↗First posted 28 Mar 2023cs.CVcited by 60

On the map: Multimodal LLMs / vision-language models → Driving VLMs and open-vocabulary detection

Abstract (arXiv)

Human-Object Interaction (HOI) detection aims to localize human-object pairs and recognize their interactions. Recently, Contrastive Language-Image Pre-training (CLIP) has shown great potential in providing interaction prior for HOI detectors via knowledge distillation. However, such approaches often rely on large-scale training data and suffer from inferior performance under few/zero-shot scenarios. In this paper, we propose a novel HOI detection framework that efficiently extracts prior knowledge from CLIP and achieves better generalization. In detail, we first introduce a novel interaction decoder to extract informative regions in the visual feature map of CLIP via a cross-attention mechanism, which is then fused with the detection backbone by a knowledge integration block for more accurate human-object pair detection. In addition, prior knowledge in CLIP text encoder is leveraged to generate a classifier by embedding HOI descriptions. To distinguish fine-grained interactions, we build a verb classifier from training data via visual semantic arithmetic and a lightweight verb representation adapter. Furthermore, we propose a training-free enhancement to exploit global HOI predictions from CLIP. Extensive experiments demonstrate that our method outperforms the state of the art by a large margin on various settings, e.g. +4.04 mAP on HICO-Det. The source code is available in https://github.com/Artanic30/HOICLIP.

Key facts. Cited by 60 papers in the map (data through 7 Oct 2026). Placed in Driving VLMs and open-vocabulary detection, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 42 of its 55 references that are in the map. First posted on arXiv 28 Mar 2023.

Builds on

42 of 55 references.

Closest ancestors · 1

Same direction, newest first

  1. 2022Learning to Prompt for Open-Vocabulary Object Detection with Vision-Language Model28 Mar

Other references · 41

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  2. 2020End-to-End Object Detection with Transformerscited by 7,384
  3. 2019Parameter-Efficient Transfer Learning for NLPcited by 4,195
  4. 2021Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervisioncited by 3,848
  5. 2021CLIP-Adapter: Better Vision-Language Models with Feature Adapterscited by 1,021
  6. 2021Open-vocabulary Object Detection via Vision and Language Knowledge Distillationcited by 903
  7. 2021Robust fine-tuning of zero-shot modelscited by 785
  8. 2019On the Efficacy of Knowledge Distillationcited by 377
  9. 2021VL-Adapter: Parameter-Efficient Transfer Learning for Vision-and-Language Taskscited by 319
  10. 2017Learning to Detect Human-Object Interactionscited by 238
  11. 2022Open-Vocabulary DETR with Conditional Matchingcited by 220
  12. 2015Visual Semantic Role Labelingcited by 188
29 more
  1. 2021Does Knowledge Distillation Really Work?cited by 187
  2. 2021HOTR: End-to-End Human-Object Interaction Detection with Transformerscited by 183
  3. 2021ZeroCap: Zero-Shot Image-to-Text Generation for Visual-Semantic Arithmeticcited by 150
  4. 2021QPIC: Query-Based Pairwise Human-Object Interaction Detection with Image-Wide Contextual Informationcited by 147
  5. 2019PPDM: Parallel Point Detection and Matching for Real-time Human-Object Interaction Detectioncited by 143
  6. 2021End-to-End Human Object Interaction Detection with HOI Transformercited by 134
  7. 2022PromptDet: Towards Open-vocabulary Detection using Uncurated Imagescited by 133
  8. 2020Visual Compositional Learning for Human-Object Interaction Detectioncited by 129
  9. 2022GEN-VLKT: Simplify Association and Enhance Interaction Understanding for HOI Detectioncited by 119
  10. 2020VSGNet: Spatial Attention Network for Detecting Human Object Interactions Using Graph Convolutionscited by 115
  11. 2021Mining the Benefits of Two-stage and One-stage HOI Detectioncited by 103
  12. 2022Exploiting Unlabeled Data with Vision and Language Models for Object Detectioncited by 97
  13. 2021Reformulating HOI Detection as Adaptive Set Predictioncited by 95
  14. 2020Detailed 2D-3D Joint Representation for Human-Object Interactioncited by 93
  15. 2020Spatially Conditioned Graphs for Detecting Human-Object Interactionscited by 93
  16. 2019Pose-aware Multi-level Feature Network for Human Object Interaction Detectioncited by 88
  17. 2020HOI Analysis: Integrating and Decomposing Human-Object Interactioncited by 81
  18. 2021Detecting Human-Object Interaction via Fabricated Compositional Learningcited by 79
  19. 2021Affordance Transfer Learning for Human-Object Interaction Detectioncited by 79
  20. 2020Detecting Human-Object Interactions with Action Co-occurrence Priorscited by 72
  21. 2022Open-Vocabulary One-Stage Detection with Hierarchical Visual-Language Knowledge Distillationcited by 67
  22. 2022RLIP: Relational Language-Image Pre-training for Human-Object Interaction Detectioncited by 54
  23. 2019Detecting Human-Object Interactions via Functional Generalizationcited by 53
  24. 2020ConsNet: Learning Consistency Graph for Zero-Shot Human-Object Interaction Detectioncited by 45
  25. 2022Mining Cross-Person Cues for Body-Part Interactiveness Learning in HOI Detectioncited by 34
  26. 2022End-to-End Zero-Shot HOI Detection via Vision and Language Knowledge Distillationcited by 30
  27. 2020A Graph-based Interactive Reasoning for Human-Object Interaction Detectioncited by 12
  28. 2022Towards Hard-Positive Query Mining for DETR-based Human-Object Interaction Detectioncited by 12
  29. 2023Weakly-supervised HOI Detection via Prior-guided Bi-level Representation Learningcited by 11

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 5; 2024: 17; 2025: 24; 2026: 14.

60 papers.

Sort:

By area:

  1. 2024From Pixels to Graphs: Open-Vocabulary Scene Graph Generation with Vision-Language Modelscited by 43
  2. 2023Detecting Any Human-Object Interaction Relationship: Universal HOI Detector with Spatial Prompt Learning on Foundation Modelscited by 34
  3. 2024Open-World Human-Object Interaction Detection via Multi-modal Promptscited by 28
  4. 2024EZ-HOI: VLM Adaptation via Guided Prompt Learning for Zero-Shot HOI Detectioncited by 28
  5. 2024Exploring Conditional Multi-Modal Prompts for Zero-shot HOI Detectioncited by 26
  6. 2024Exploring the Potential of Large Foundation Models for Open-Vocabulary HOI Detectioncited by 25
  7. 2023Boosting Human-Object Interaction Detection with Text-to-Image Diffusion Modelcited by 21
  8. 2024Human-Object Interaction Detection Collaborated with Large Relation-driven Diffusion Modelscited by 15
  9. 2024Nonverbal Interaction Detectioncited by 14
  10. 2024INTRA: Interaction Relationship-aware Weakly Supervised Affordance Groundingcited by 12
  11. 2025Locality-Aware Zero-Shot Human-Object Interaction Detectioncited by 12
  12. 2025HOLa: Zero-Shot HOI Detection with Low-Rank Decomposed VLM Feature Adaptationcited by 12
38 more
  1. 2025Open-Vocabulary HOI Detection with Interaction-aware Prompt and Concept Calibrationcited by 7
  2. 2024Enhancing Human-Centered Dynamic Scene Understanding via Multiple LLMs Collaborated Reasoningcited by 6
  3. 2026What if Agents Could Imagine? Reinforcing Open-Vocabulary HOI Comprehension through Generationcited by 6
  4. 2023Exploiting CLIP for Zero-shot HOI Detection Requires Knowledge Distillation at Multiple Levelscited by 5
  5. 2024CycleHOI: Improving Human-Object Interaction Detection with Cycle Consistency of Detection and Generationcited by 5
  6. 2024A Review of Human-Object Interaction Detectioncited by 5
  7. 2025ANNEXE: Unified Analyzing, Answering, and Pixel Grounding for Egocentric Interactioncited by 5
  8. 2025CrossHOI-Bench: A Unified Benchmark for HOI Evaluation across Vision-Language Models and HOI-Specific Methodscited by 5
  9. 2025Learning Human-Object Interaction as Groupscited by 5
  10. 2023Focusing on what to decode and what to train: SOV Decoding with Specific Target Guided DeNoising and Vision Language Advisorcited by 4
  11. 2025VHOI: Controllable Video Generation of Human-Object Interactions from Sparse Trajectories via Motion Densificationcited by 4
  12. 2023Enhancing HOI Detection with Contextual Cues from Large Vision-Language Modelscited by 3
  13. 2024Towards Zero-shot Human-Object Interaction Detection via Vision-Language Integrationcited by 3
  14. 2024CoLA: Conditional Dropout and Language-driven Robust Dual-modal Salient Object Detectioncited by 3
  15. 2025SGC-Net: Stratified Granular Comparison Network for Open-Vocabulary HOI Detectioncited by 3
  16. 2025Egocentric Human-Object Interaction Detection: A New Benchmark and Methodcited by 3
  17. 2025No More Sibling Rivalry: Debiasing Human-Object Interaction Detectioncited by 3
  18. 2025Visual Diversity and Region-aware Prompt Learning for Zero-shot HOI Detectioncited by 3
  19. 2026DA-DPO: Cost-efficient Difficulty-aware Preference Optimization for Reducing MLLM Hallucinationscited by 3
  20. 2024Geometric Features Enhanced Human-Object Interaction Detectioncited by 2
  21. 2026Streamlined Open-Vocabulary Human-Object Interaction Detectioncited by 2
  22. 2024VLM-HOI: Vision Language Models for Interpretable Human-Object Interaction Analysiscited by 1
  23. 2025Dynamic Scene Understanding from Vision-Language Representationscited by 1
  24. 2025DQEN: Dual Query Enhancement Network for DETR-based HOI Detectioncited by 1
  25. 2025Trainee Action Recognition through Interaction Analysis in CCATT Mixed-Reality Trainingcited by 1
  26. 2025Mitigating Long-Tail Bias in HOI Detection via Adaptive Diversity Cachecited by 1
  27. 2026Zero-shot HOI Detection with MLLM-based Detector-agnostic Interaction Recognitioncited by 1
  28. 2026WikiCLIP: An Efficient Contrastive Baseline for Open-domain Visual Entity Recognitioncited by 1
  29. 2026SportSkills: Physical Skill Learning from Sports Instructional Videoscited by 1
  30. 2026SHOE: Semantic HOI Open-Vocabulary Evaluation Metriccited by 1
  31. 2026Mining Instance-Centric Vision-Language Contexts for Human-Object Interaction Detectioncited by 1
  32. 2026Unleashing Multimodal Large Language Models for Training-free HOI Detection in the Wildcited by 1
  33. 2024FreeA: Human-object Interaction Detection using Free Annotation Labelscited by 0
  34. 2024Exploring Interactive Semantic Alignment for Efficient HOI Detection with Vision-language Modelcited by 0
  35. 2025B-RIGHT: Benchmark Re-evaluation for Integrity in Generalized Human-Object Interaction Testingcited by 0
  36. 2025On the Robustness of Human-Object Interaction Detection against Distribution Shiftcited by 0
  37. 2025Funnel-HOI: Top-Down Perception for Zero-Shot HOI Detectioncited by 0
  38. 2025HOID-R1: Reinforcement Learning for Open-World Human-Object Interaction Detection Reasoning with Multimodal Large Language Modelcited by 0

50 most cited of 60 All 60 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2024CL-HOI: Cross-Level Human-Object Interaction Distillation from Vision Large Language Models
  2. 2026Harnessing Multimodal Large Language Models for Training-Free Human-Object Interaction Detection
  3. 2025Dynamic Scoring with Enhanced Semantics for Training-Free Human-Object Interaction Detection
  4. 2023Weakly-Supervised HOI Detection from Interaction Labels Only and Language/Vision-Language Priors
  5. 2025Bilateral Collaboration with Large Vision-Language Models for Open Vocabulary Human-Object Interaction Detection
  6. 2026FineHOI: Part-Aware Dense Representations for Zero-Shot Human-Object Interaction Detection
  7. 2023Efficient Adaptive Human-Object Interaction Detection with Concept-guided Memory
  8. 2023Neural-Logic Human-Object Interaction Detection