mlsanityData to 7 Oct 2026

Unseen No More: Unlocking the Potential of CLIP for Generative Zero-shot HOI Detection

arXiv 2408.05974 ↗First posted 12 Aug 2024cs.CVcited by 11

Not on the map. Why?

Abstract (arXiv)

Zero-shot human-object interaction (HOI) detector is capable of generalizing to HOI categories even not encountered during training. Inspired by the impressive zero-shot capabilities offered by CLIP, latest methods strive to leverage CLIP embeddings for improving zero-shot HOI detection. However, these embedding-based methods train the classifier on seen classes only, inevitably resulting in seen-unseen confusion for the model during inference. Besides, we find that using prompt-tuning and adapters further increases the gap between seen and unseen accuracy. To tackle this challenge, we present the first generation-based model using CLIP for zero-shot HOI detection, coined HOIGen. It allows to unlock the potential of CLIP for feature generation instead of feature extraction only. To achieve it, we develop a CLIP-injected feature generator in accordance with the generation of human, object and union features. Then, we extract realistic features of seen samples and mix them with synthetic features together, allowing the model to train seen and unseen classes jointly. To enrich the HOI scores, we construct a generative prototype bank in a pairwise HOI recognition branch, and a multi-knowledge prototype bank in an image-wise HOI recognition branch, respectively. Extensive experiments on HICO-DET benchmark demonstrate our HOIGen achieves superior performance for both seen and unseen classes under various zero-shot settings, compared with other top-performing methods. Code is available at: https://github.com/soberguo/HOIGen

Key facts. Cited by 11 papers in the map (data through 7 Oct 2026). Builds on 36 of its 53 references that are in the map. First posted on arXiv 12 Aug 2024.

Builds on

36 of 53 references.

References · 36

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  2. 2020End-to-End Object Detection with Transformerscited by 7,384
  3. 2017Mask R-CNNcited by 7,259
  4. 2021Emerging Properties in Self-Supervised Vision Transformerscited by 6,893
  5. 2021Learning to Prompt for Vision-Language Modelscited by 2,269
  6. 2017Improved Training of Wasserstein GANscited by 2,223
  7. 2021CLIP-Adapter: Better Vision-Language Models with Feature Adapterscited by 1,021
  8. 2015Autoencoding beyond pixels using a learned similarity metriccited by 407
  9. 2022Tip-Adapter: Training-free Adaption of CLIP for Few-shot Classificationcited by 355
  10. 2018Generalized Zero- and Few-Shot Learning via Aligned Variational Autoencoderscited by 206
  11. 2023Prompt, Generate, then Cache: Cascade of Foundation Models makes Strong Few-shot Learnerscited by 152
  12. 2021QPIC: Query-Based Pairwise Human-Object Interaction Detection with Image-Wide Contextual Informationcited by 147
24 more
  1. 2018iCAN: Instance-Centric Attention Network for Human-Object Interaction Detectioncited by 126
  2. 2022GEN-VLKT: Simplify Association and Enhance Interaction Understanding for HOI Detectioncited by 119
  3. 2020DRG: Dual Relation Graph for Human-Object Interaction Detectioncited by 117
  4. 2021Reformulating HOI Detection as Adaptive Set Predictioncited by 95
  5. 2021Efficient Two-Stage Detection of Human-Object Interactions with a Novel Unary-Pairwise Transformercited by 91
  6. 2020HOI Analysis: Integrating and Decomposing Human-Object Interactioncited by 81
  7. 2021Detecting Human-Object Interaction via Fabricated Compositional Learningcited by 79
  8. 2022CLIP-GEN: Language-Free Training of a Text-to-Image Generator with CLIPcited by 60
  9. 2019Dont Even Look Once: Synthesizing Features for Zero-Shot Detectioncited by 58
  10. 2023Exploring Predicate Visual Context in Detecting Human-Object Interactionscited by 56
  11. 2022AvatarCLIP: Zero-Shot Text-Driven Generation and Animation of 3D Avatarscited by 55
  12. 2019Detecting Human-Object Interactions via Functional Generalizationcited by 53
  13. 2023Improving Zero-Shot Generalization for CLIP with Synthesized Promptscited by 51
  14. 2020ConsNet: Learning Consistency Graph for Zero-Shot Human-Object Interaction Detectioncited by 45
  15. 2022Being Comes from Not-being: Open-vocabulary Text-to-Motion Generation with Wordless Trainingcited by 45
  16. 2022Human-Object Interaction Detection via Disentangled Transformercited by 40
  17. 2022What to look at and where: Semantic and Spatial Refined Transformer for detecting human-object interactionscited by 35
  18. 2023Detecting Any Human-Object Interaction Relationship: Universal HOI Detector with Spatial Prompt Learning on Foundation Modelscited by 34
  19. 2022End-to-End Zero-Shot HOI Detection via Vision and Language Knowledge Distillationcited by 30
  20. 2021Entropy-Based Uncertainty Calibration for Generalized Zero-Shot Learningcited by 8
  21. 2022Distance Matters in Human-Object Interaction Detectioncited by 8
  22. 2021Cross Knowledge-based Generative Zero-Shot Learning Approach with Taxonomy Regularizationcited by 5
  23. 2023Exploiting CLIP for Zero-shot HOI Detection Requires Knowledge Distillation at Multiple Levelscited by 5
  24. 2023Improving Human-Object Interaction Detection via Virtual Image Learningcited by 4

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 7; 2026: 4.

11 papers.

  1. 2025Open-Vocabulary HOI Detection with Interaction-aware Prompt and Concept Calibrationcited by 7
  2. 2026What if Agents Could Imagine? Reinforcing Open-Vocabulary HOI Comprehension through Generationcited by 6
  3. 2025Differential Contrastive Training for Gaze Estimationcited by 3
  4. 2025DQEN: Dual Query Enhancement Network for DETR-based HOI Detectioncited by 1
  5. 2025Mitigating Long-Tail Bias in HOI Detection via Adaptive Diversity Cachecited by 1
  6. 2026Unleashing Multimodal Large Language Models for Training-free HOI Detection in the Wildcited by 1
  7. 2025Multimodal Graph Network Modeling for Human-Object Interaction Detection with PDE Graph Diffusioncited by 0
  8. 2025Contextualized Representation Learning for Effective Human-Object Interaction Detectioncited by 0
  9. 2025Generative Human-Object Interaction Detection via Differentiable Cognitive Steering of Multi-modal LLMscited by 0
  10. 2026Towards Unconstrained Human-Object Interactioncited by 0
  11. 2026FineHOI: Part-Aware Dense Representations for Zero-Shot Human-Object Interaction Detectioncited by 0