mlsanityData to 7 Oct 2026

FreeA: Human-object Interaction Detection using Free Annotation Labels

arXiv 2403.01840 ↗First posted 4 Mar 2024cs.CV cs.AIcited by 0

Not on the map. Why?

Abstract (arXiv)

Recent human-object interaction (HOI) detection methods depend on extensively annotated image datasets, which require a significant amount of manpower. In this paper, we propose a novel self-adaptive, language-driven HOI detection method, termed FreeA. This method leverages the adaptability of the text-image model to generate latent HOI labels without requiring manual annotation. Specifically, FreeA aligns image features of human-object pairs with HOI text templates and employs a knowledge-based masking technique to decrease improbable interactions. Furthermore, FreeA implements a proposed method for matching interaction correlations to increase the probability of actions associated with a particular action, thereby improving the generated HOI labels. Experiments on two benchmark datasets showcase that FreeA achieves state-of-the-art performance among weakly supervised HOI competitors. Our proposal gets +13.29 (159%uparrow) mAP and +17.30 (98%uparrow) mAP than the newest ``Weakly'' supervised model, and +7.19 (28%uparrow) mAP and +14.69 (34%uparrow) mAP than the latest ``Weakly+'' supervised model, respectively, on HICO-DET and V-COCO datasets, more accurate in localizing and classifying the interactive actions. The source code will be made public.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Builds on 33 of its 40 references that are in the map. First posted on arXiv 4 Mar 2024.

Builds on

33 of 40 references.

References · 33

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  2. 2015Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networkscited by 9,617
  3. 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,069
  4. 2022BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generationcited by 4,310
  5. 2015Fast R-CNNcited by 2,930
  6. 2017Learning to Detect Human-Object Interactionscited by 238
  7. 2017Detecting and Recognizing Human-Object Interactionscited by 199
  8. 2021QPIC: Query-Based Pairwise Human-Object Interaction Detection with Image-Wide Contextual Informationcited by 147
  9. 2019PPDM: Parallel Point Detection and Matching for Real-time Human-Object Interaction Detectioncited by 143
  10. 2021End-to-End Human Object Interaction Detection with HOI Transformercited by 134
  11. 2018iCAN: Instance-Centric Attention Network for Human-Object Interaction Detectioncited by 126
  12. 2022GEN-VLKT: Simplify Association and Enhance Interaction Understanding for HOI Detectioncited by 119
21 more
  1. 2020VSGNet: Spatial Attention Network for Detecting Human Object Interactions Using Graph Convolutionscited by 115
  2. 2020Detailed 2D-3D Joint Representation for Human-Object Interactioncited by 93
  3. 2020Spatially Conditioned Graphs for Detecting Human-Object Interactionscited by 93
  4. 2019Pose-aware Multi-level Feature Network for Human Object Interaction Detectioncited by 88
  5. 2020HOI Analysis: Integrating and Decomposing Human-Object Interactioncited by 81
  6. 2021Detecting Human-Object Interaction via Fabricated Compositional Learningcited by 79
  7. 2021Affordance Transfer Learning for Human-Object Interaction Detectioncited by 79
  8. 2023HOICLIP: Efficient Knowledge Transfer for HOI Detection with Vision-Language Modelscited by 60
  9. 2023Exploring Predicate Visual Context in Detecting Human-Object Interactionscited by 56
  10. 2020ConsNet: Learning Consistency Graph for Zero-Shot Human-Object Interaction Detectioncited by 45
  11. 2023RLIPv2: Fast Scaling of Relational Language-Image Pre-trainingcited by 44
  12. 2023Relational Context Learning for Human-Object Interaction Detectioncited by 35
  13. 2017PPR-FCN: Weakly Supervised Visual Relation Detection via Parallel Pairwise R-FCNcited by 25
  14. 2021Improving Human-Object Interaction Detection via Phrase Learning and Label Compositioncited by 15
  15. 2020Explanation-based Weakly-supervised Learning of Visual Relations with Graph Networkscited by 13
  16. 2021ACP++: Action Co-occurrence Priors for Human-Object Interaction Detectioncited by 11
  17. 2023Weakly-supervised HOI Detection via Prior-guided Bi-level Representation Learningcited by 11
  18. 2020Detecting Human-Object Interaction with Mixed Supervisioncited by 9
  19. 2021GTNet:Guided Transformer Network for Detecting Human-Object Interactionscited by 8
  20. 2021Human-Object Interaction Detection via Weak Supervisioncited by 5
  21. 2019Semantics to Space(S2S): Embedding semantics into spatial space for zero-shot verb-object query inferencingcited by 2

Built on by

Not cited yet.