mlsanityData to 14 Oct 2026

Re-mine, Learn and Reason: Exploring the Cross-modal Semantic Correlations for Language-guided HOI detection

arXiv 2307.13529 ↗First posted 25 Jul 2023cs.CV cs.AIcited by 13

Not on the map. Why?

Abstract (arXiv)

Human-Object Interaction (HOI) detection is a challenging computer vision task that requires visual models to address the complex interactive relationship between humans and objects and predict HOI triplets. Despite the challenges posed by the numerous interaction combinations, they also offer opportunities for multimodal learning of visual texts. In this paper, we present a systematic and unified framework (RmLR) that enhances HOI detection by incorporating structured text knowledge. Firstly, we qualitatively and quantitatively analyze the loss of interaction information in the two-stage HOI detector and propose a re-mining strategy to generate more comprehensive visual representation.Secondly, we design more fine-grained sentence- and word-level alignment and knowledge transfer strategies to effectively address the many-to-many matching problem between multiple interactions and multiple texts.These strategies alleviate the matching confusion problem that arises when multiple interactions occur simultaneously, thereby improving the effectiveness of the alignment process. Finally, HOI reasoning by visual features augmented with textual knowledge substantially improves the understanding of interactions. Experimental results illustrate the effectiveness of our approach, where state-of-the-art performance is achieved on public benchmarks. We further analyze the effects of different components of our approach to provide insights into its efficacy.

Key facts. Cited by 13 papers in the map (data through 14 Oct 2026). Builds on 60 of its 71 references that are in the map. First posted on arXiv 25 Jul 2023.

Builds on

60 of 71 references.

References · 60

  1. 2015Deep Residual Learning for Image Recognitioncited by 52,096
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  3. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  4. 2014Microsoft COCO: Common Objects in Contextcited by 17,608
  5. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,372
  6. 2020End-to-End Object Detection with Transformerscited by 7,414
  7. 2022Flamingo: a Visual Language Model for Few-Shot Learningcited by 4,386
  8. 2019ALBERT: A Lite BERT for Self-supervised Learning of Language Representationscited by 2,790
  9. 2016Google's Neural Machine Translation System: Bridging the Gap between Human and Machine Translationcited by 1,475
  10. 2021Grounded Language-Image Pre-trainingcited by 1,156
  11. 2017Vision-and-Language Navigation: Interpreting visually-grounded navigation instructions in real environmentscited by 1,027
  12. 2017Multimodal Machine Learning: A Survey and Taxonomycited by 1,013
38 more
  1. 2021Multimodal Few-Shot Learning with Frozen Language Modelscited by 661
  2. 2020MobileBERT: a Compact Task-Agnostic BERT for Resource-Limited Devicescited by 482
  3. 2021SLIP: Self-supervision meets Language-Image Pre-trainingcited by 436
  4. 2022GLIPv2: Unifying Localization and Vision-Language Understandingcited by 275
  5. 2017Learning to Detect Human-Object Interactionscited by 240
  6. 2018CrowdPose: Efficient Crowded Scenes Pose Estimation and A New Benchmarkcited by 214
  7. 2017Detecting and Recognizing Human-Object Interactionscited by 199
  8. 2015Visual Semantic Role Labelingcited by 190
  9. 2021HOTR: End-to-End Human-Object Interaction Detection with Transformerscited by 183
  10. 2018Learning Human-Object Interactions by Graph Parsing Neural Networkscited by 176
  11. 2021QPIC: Query-Based Pairwise Human-Object Interaction Detection with Image-Wide Contextual Informationcited by 147
  12. 2019PPDM: Parallel Point Detection and Matching for Real-time Human-Object Interaction Detectioncited by 143
  13. 2021End-to-End Human Object Interaction Detection with HOI Transformercited by 135
  14. 2020Visual Compositional Learning for Human-Object Interaction Detectioncited by 130
  15. 2018iCAN: Instance-Centric Attention Network for Human-Object Interaction Detectioncited by 127
  16. 2022GEN-VLKT: Simplify Association and Enhance Interaction Understanding for HOI Detectioncited by 120
  17. 2020DRG: Dual Relation Graph for Human-Object Interaction Detectioncited by 117
  18. 2020VSGNet: Spatial Attention Network for Detecting Human Object Interactions Using Graph Convolutionscited by 115
  19. 2021Mining the Benefits of Two-stage and One-stage HOI Detectioncited by 103
  20. 2021Reformulating HOI Detection as Adaptive Set Predictioncited by 95
  21. 2020Detailed 2D-3D Joint Representation for Human-Object Interactioncited by 93
  22. 2020Spatially Conditioned Graphs for Detecting Human-Object Interactionscited by 93
  23. 2021Efficient Two-Stage Detection of Human-Object Interactions with a Novel Unary-Pairwise Transformercited by 91
  24. 2019Pose-aware Multi-level Feature Network for Human Object Interaction Detectioncited by 88
  25. 2020HOI Analysis: Integrating and Decomposing Human-Object Interactioncited by 81
  26. 2021Detecting Human-Object Interaction via Fabricated Compositional Learningcited by 79
  27. 2021Affordance Transfer Learning for Human-Object Interaction Detectioncited by 79
  28. 2021MDMMT: Multidomain Multimodal Transformer for Video Retrievalcited by 78
  29. 2023Can GPT-4 Perform Neural Architecture Search?cited by 74
  30. 2020Detecting Human-Object Interactions with Action Co-occurrence Priorscited by 72
  31. 2021Glance and Gaze: Inferring Action-aware Points for One-Stage Human-Object Interaction Detectioncited by 58
  32. 2022RLIP: Relational Language-Image Pre-training for Human-Object Interaction Detectioncited by 55
  33. 2019Detecting Human-Object Interactions via Functional Generalizationcited by 53
  34. 2022Align, Reason and Learn: Enhancing Medical Vision-and-Language Pre-training with Knowledgecited by 51
  35. 2020ConsNet: Learning Consistency Graph for Zero-Shot Human-Object Interaction Detectioncited by 46
  36. 2022Interactiveness Field in Human-Object Interactionscited by 40
  37. 2022What to look at and where: Semantic and Spatial Refined Transformer for detecting human-object interactionscited by 35
  38. 2021Prioritized Architecture Sampling with Monto-Carlo Tree Searchcited by 32

50 most cited of 60 All 60 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 2; 2024: 5; 2025: 4; 2026: 2.

13 papers.

Sort:

By area:

  1. 2023Detecting Any Human-Object Interaction Relationship: Universal HOI Detector with Spatial Prompt Learning on Foundation Modelscited by 34
  2. 2024Open-World Human-Object Interaction Detection via Multi-modal Promptscited by 28
  3. 2024Exploring the Potential of Large Foundation Models for Open-Vocabulary HOI Detectioncited by 25
  4. 2024Human-Object Interaction Detection Collaborated with Large Relation-driven Diffusion Modelscited by 15
  5. 2025Open-Vocabulary HOI Detection with Interaction-aware Prompt and Concept Calibrationcited by 8
  6. 2024Exploring Self- and Cross-Triplet Correlations for Human-Object Interaction Detectioncited by 7
  7. 2023DiffNAS: Bootstrapping Diffusion Models by Prompting for Better Architecturescited by 6
  8. 2024Towards Zero-shot Human-Object Interaction Detection via Vision-Language Integrationcited by 4
  9. 2025SGC-Net: Stratified Granular Comparison Network for Open-Vocabulary HOI Detectioncited by 3
  10. 2025HOID-R1: Reinforcement Learning for Open-World Human-Object Interaction Detection Reasoning with Multimodal Large Language Modelcited by 2
  11. 2026Mining Instance-Centric Vision-Language Contexts for Human-Object Interaction Detectioncited by 1
  12. 2026Unleashing Multimodal Large Language Models for Training-free HOI Detection in the Wildcited by 1
1 more
  1. 2025Generative Human-Object Interaction Detection via Differentiable Cognitive Steering of Multi-modal LLMscited by 0