mlsanityData to 7 Oct 2026

GEN-VLKT: Simplify Association and Enhance Interaction Understanding for HOI Detection

arXiv 2203.13954 ↗First posted 26 Mar 2022cs.CV cs.AIcited by 119

Not on the map. Why?

Abstract (arXiv)

The task of Human-Object Interaction~(HOI) detection could be divided into two core problems, i.e., human-object association and interaction understanding. In this paper, we reveal and address the disadvantages of the conventional query-driven HOI detectors from the two aspects. For the association, previous two-branch methods suffer from complex and costly post-matching, while single-branch methods ignore the features distinction in different tasks. We propose Guided-Embedding Network~(GEN) to attain a two-branch pipeline without post-matching. In GEN, we design an instance decoder to detect humans and objects with two independent query sets and a position Guided Embedding~(p-GE) to mark the human and object in the same position as a pair. Besides, we design an interaction decoder to classify interactions, where the interaction queries are made of instance Guided Embeddings (i-GE) generated from the outputs of each instance decoder layer. For the interaction understanding, previous methods suffer from long-tailed distribution and zero-shot discovery. This paper proposes a Visual-Linguistic Knowledge Transfer (VLKT) training strategy to enhance interaction understanding by transferring knowledge from a visual-linguistic pre-trained model CLIP. In specific, we extract text embeddings for all labels with CLIP to initialize the classifier and adopt a mimic loss to minimize the visual feature distance between GEN and CLIP. As a result, GEN-VLKT outperforms the state of the art by large margins on multiple datasets, e.g., +5.05 mAP on HICO-Det. The source codes are available at https://github.com/YueLiao/gen-vlkt.

Key facts. Cited by 119 papers in the map (data through 7 Oct 2026). Builds on 38 of its 47 references that are in the map. First posted on arXiv 26 Mar 2022.

Builds on

38 of 47 references.

References · 38

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  2. 2020End-to-End Object Detection with Transformerscited by 7,384
  3. 2019ViLBERT: Pretraining Task-Agnostic Visiolinguistic Representations for Vision-and-Language Taskscited by 2,219
  4. 2019VL-BERT: Pre-training of Generic Visual-Linguistic Representationscited by 954
  5. 2021Open-vocabulary Object Detection via Vision and Language Knowledge Distillationcited by 903
  6. 2017Learning to Detect Human-Object Interactionscited by 238
  7. 2017Detecting and Recognizing Human-Object Interactionscited by 199
  8. 2015Visual Semantic Role Labelingcited by 188
  9. 2021HOTR: End-to-End Human-Object Interaction Detection with Transformerscited by 183
  10. 2018Learning Human-Object Interactions by Graph Parsing Neural Networkscited by 176
  11. 2021QPIC: Query-Based Pairwise Human-Object Interaction Detection with Image-Wide Contextual Informationcited by 147
  12. 2019PPDM: Parallel Point Detection and Matching for Real-time Human-Object Interaction Detectioncited by 143
26 more
  1. 2021End-to-End Human Object Interaction Detection with HOI Transformercited by 134
  2. 2020Visual Compositional Learning for Human-Object Interaction Detectioncited by 129
  3. 2018iCAN: Instance-Centric Attention Network for Human-Object Interaction Detectioncited by 126
  4. 2020DRG: Dual Relation Graph for Human-Object Interaction Detectioncited by 117
  5. 2020VSGNet: Spatial Attention Network for Detecting Human Object Interactions Using Graph Convolutionscited by 115
  6. 2020Learning Human-Object Interaction Detection using Interaction Pointscited by 108
  7. 2021Mining the Benefits of Two-stage and One-stage HOI Detectioncited by 103
  8. 2021Reformulating HOI Detection as Adaptive Set Predictioncited by 95
  9. 2020Detailed 2D-3D Joint Representation for Human-Object Interactioncited by 93
  10. 2020Spatially Conditioned Graphs for Detecting Human-Object Interactionscited by 93
  11. 2019Pose-aware Multi-level Feature Network for Human Object Interaction Detectioncited by 88
  12. 2020HOI Analysis: Integrating and Decomposing Human-Object Interactioncited by 81
  13. 2021Detecting Human-Object Interaction via Fabricated Compositional Learningcited by 79
  14. 2021Affordance Transfer Learning for Human-Object Interaction Detectioncited by 79
  15. 2020Detecting Human-Object Interactions with Action Co-occurrence Priorscited by 72
  16. 2021Glance and Gaze: Inferring Action-aware Points for One-Stage Human-Object Interaction Detectioncited by 58
  17. 2019Detecting Human-Object Interactions via Functional Generalizationcited by 53
  18. 2020Contextual Heterogeneous Graph Network for Human-Object Interaction Detectioncited by 50
  19. 2020ConsNet: Learning Consistency Graph for Zero-Shot Human-Object Interaction Detectioncited by 45
  20. 2021Visual Relationship Detection Using Part-and-Sum Transformers with Composite Queriescited by 31
  21. 2021Exploiting Scene Graphs for Human-Object Interaction Detectioncited by 22
  22. 2021Improving Human-Object Interaction Detection via Phrase Learning and Label Compositioncited by 15
  23. 2018Transferable Interactiveness Knowledge for Human-Object Interaction Detectioncited by 12
  24. 2020A Graph-based Interactive Reasoning for Human-Object Interaction Detectioncited by 12
  25. 2018No-Frills Human-Object Interaction Detection: Factorization, Layout Encodings, and Training Techniquescited by 8
  26. 2021Is Object Detection Necessary for Human-Object Interaction Recognition?cited by 3

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 5; 2023: 36; 2024: 32; 2025: 31; 2026: 15.

119 papers.

Sort:

By area:

  1. 2022Full-Body Articulated Human-Object Interactioncited by 76
  2. 2023Object-Aware Distillation Pyramid for Open-Vocabulary Object Detectioncited by 63
  3. 2023HOICLIP: Efficient Knowledge Transfer for HOI Detection with Vision-Language Modelscited by 60
  4. 2023Exploring Predicate Visual Context in Detecting Human-Object Interactionscited by 56
  5. 2022RLIP: Relational Language-Image Pre-training for Human-Object Interaction Detectioncited by 54
  6. 2023RLIPv2: Fast Scaling of Relational Language-Image Pre-trainingcited by 44
  7. 2024From Pixels to Graphs: Open-Vocabulary Scene Graph Generation with Vision-Language Modelscited by 43
  8. 2023VL-SAT: Visual-Linguistic Semantics Assisted Training for 3D Semantic Scene Graph Prediction in Point Cloudcited by 42
  9. 2023ViPLO: Vision Transformer based Pose-Conditioned Self-Loop Graph for Human-Object Interaction Detectioncited by 41
  10. 2023Efficient Adaptive Human-Object Interaction Detection with Concept-guided Memorycited by 38
  11. 2023Detecting Any Human-Object Interaction Relationship: Universal HOI Detector with Spatial Prompt Learning on Foundation Modelscited by 34
  12. 2022End-to-End Zero-Shot HOI Detection via Vision and Language Knowledge Distillationcited by 30
38 more
  1. 2024Open-World Human-Object Interaction Detection via Multi-modal Promptscited by 28
  2. 2024EZ-HOI: VLM Adaptation via Guided Prompt Learning for Zero-Shot HOI Detectioncited by 28
  3. 2023Zero-shot Visual Relation Detection via Composite Visual Cues from Large Language Modelscited by 27
  4. 2024Exploring Conditional Multi-Modal Prompts for Zero-shot HOI Detectioncited by 26
  5. 2023Neural-Logic Human-Object Interaction Detectioncited by 25
  6. 2024Exploring the Potential of Large Foundation Models for Open-Vocabulary HOI Detectioncited by 25
  7. 2024F-HOI: Toward Fine-grained Semantic-Aligned 3D Human-Object Interactionscited by 24
  8. 2023Category Query Learning for Human-Object Interaction Classificationcited by 21
  9. 2023Boosting Human-Object Interaction Detection with Text-to-Image Diffusion Modelcited by 21
  10. 2023Zero-shot Referring Expression Comprehension via Structural Similarity Between Images and Captionscited by 21
  11. 2023CoVLM: Composing Visual Entities and Relationships in Large Language Models Via Communicative Decodingcited by 20
  12. 2024Learning from Observer Gaze:Zero-Shot Attention Prediction Oriented by Human-Object Interaction Recognitioncited by 17
  13. 2023Towards a Unified Transformer-based Framework for Scene Graph Generation and Human-object Interaction Detectioncited by 15
  14. 2024Human-Object Interaction Detection Collaborated with Large Relation-driven Diffusion Modelscited by 15
  15. 2024Nonverbal Interaction Detectioncited by 14
  16. 2023Unified Visual Relationship Detection with Vision and Language Modelscited by 13
  17. 2023Re-mine, Learn and Reason: Exploring the Cross-modal Semantic Correlations for Language-guided HOI detectioncited by 13
  18. 2022Distilling Vision-Language Pre-training to Collaborate with Weakly-Supervised Temporal Action Localizationcited by 12
  19. 2023HOI4ABOT: Human-Object Interaction Anticipation for Human Intention Reading Collaborative roBOTscited by 12
  20. 2025Locality-Aware Zero-Shot Human-Object Interaction Detectioncited by 12
  21. 2025HOLa: Zero-Shot HOI Detection with Low-Rank Decomposed VLM Feature Adaptationcited by 12
  22. 2023Weakly-supervised HOI Detection via Prior-guided Bi-level Representation Learningcited by 11
  23. 2023Diagnosing Human-object Interaction Detectorscited by 11
  24. 2023LLMs as Visual Explainers: Advancing Image Classification with Evolving Visual Descriptionscited by 11
  25. 2024Groupwise Query Specialization and Quality-Aware Multi-Assignment for Transformer-based Visual Relationship Detectioncited by 11
  26. 2024Unseen No More: Unlocking the Potential of CLIP for Generative Zero-shot HOI Detectioncited by 11
  27. 2024Hydra-SGG: Hybrid Relation Assignment for One-stage Scene Graph Generationcited by 10
  28. 2023HODN: Disentangling Human-Object Feature for HOI Detectioncited by 9
  29. 2023Agglomerative Transformer for Human-Object Interaction Detectioncited by 8
  30. 2024Exploring Self- and Cross-Triplet Correlations for Human-Object Interaction Detectioncited by 7
  31. 2025Open-Vocabulary HOI Detection with Interaction-aware Prompt and Concept Calibrationcited by 7
  32. 2023Weakly-Supervised HOI Detection from Interaction Labels Only and Language/Vision-Language Priorscited by 6
  33. 2024Tri-modal Confluence with Temporal Dynamics for Scene Graph Generation in Operating Roomscited by 6
  34. 2024A Plug-and-Play Method for Rare Human-Object Interactions Detection by Bridging Domain Gapcited by 6
  35. 2026What if Agents Could Imagine? Reinforcing Open-Vocabulary HOI Comprehension through Generationcited by 6
  36. 2023Exploiting CLIP for Zero-shot HOI Detection Requires Knowledge Distillation at Multiple Levelscited by 5
  37. 2024Weakly-Supervised 3D Scene Graph Generation via Visual-Linguistic Assisted Pseudo-labelingcited by 5
  38. 2024CycleHOI: Improving Human-Object Interaction Detection with Cycle Consistency of Detection and Generationcited by 5

50 most cited of 119 All 119 (JSON)