mlsanityData to 14 Oct 2026

Contextualized Representation Learning for Effective Human-Object Interaction Detection

arXiv 2509.12784 ↗First posted 16 Sep 2025cs.CVcited by 0

Not on the map. Why?

Abstract (arXiv)

Human-Object Interaction (HOI) detection aims to simultaneously localize human-object pairs and recognize their interactions. While recent two-stage approaches have made significant progress, they still face challenges due to incomplete context modeling. In this work, we introduce a Contextualized Representation Learning that integrates both affordance-guided reasoning and contextual prompts with visual cues to better capture complex interactions. We enhance the conventional HOI detection framework by expanding it beyond simple human-object pairs to include multivariate relationships involving auxiliary entities like tools. Specifically, we explicitly model the functional role (affordance) of these auxiliary objects through triplet structures <human, tool, object>. This enables our model to identify tool-dependent interactions such as 'filling'. Furthermore, the learnable prompt is enriched with instance categories and subsequently integrated with contextual visual features using an attention mechanism. This process aligns language with image content at both global and regional levels. These contextualized representations equip the model with enriched relational cues for more reliable reasoning over complex, context-dependent interactions. Our proposed method demonstrates superior performance on both the HICO-Det and V-COCO datasets in most scenarios. The source code is available at https://github.com/lzzhhh1019/CRL.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 36 of its 49 references that are in the map. First posted on arXiv 16 Sep 2025.

Builds on

36 of 49 references.

References · 36

  1. 2015Deep Residual Learning for Image Recognitioncited by 52,096
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  3. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  4. 2020Language Models are Few-Shot Learnerscited by 32,419
  5. 2014Microsoft COCO: Common Objects in Contextcited by 17,608
  6. 2020End-to-End Object Detection with Transformerscited by 7,414
  7. 2021Learning to Prompt for Vision-Language Modelscited by 2,281
  8. 2022Conditional Prompt Learning for Vision-Language Modelscited by 1,433
  9. 2017Learning to Detect Human-Object Interactionscited by 240
  10. 2015Visual Semantic Role Labelingcited by 190
  11. 2021HOTR: End-to-End Human-Object Interaction Detection with Transformerscited by 183
  12. 2021QPIC: Query-Based Pairwise Human-Object Interaction Detection with Image-Wide Contextual Informationcited by 147
24 more
  1. 2021End-to-End Human Object Interaction Detection with HOI Transformercited by 135
  2. 2022GEN-VLKT: Simplify Association and Enhance Interaction Understanding for HOI Detectioncited by 120
  3. 2021Mining the Benefits of Two-stage and One-stage HOI Detectioncited by 103
  4. 2021Efficient Two-Stage Detection of Human-Object Interactions with a Novel Unary-Pairwise Transformercited by 91
  5. 2021Affordance Transfer Learning for Human-Object Interaction Detectioncited by 79
  6. 2023HOICLIP: Efficient Knowledge Transfer for HOI Detection with Vision-Language Modelscited by 61
  7. 2023Exploring Predicate Visual Context in Detecting Human-Object Interactionscited by 56
  8. 2022Exploring Structure-aware Transformer over Interaction Proposals for Human-Object Interaction Detectioncited by 47
  9. 2022Interactiveness Field in Human-Object Interactionscited by 40
  10. 2023Efficient Adaptive Human-Object Interaction Detection with Concept-guided Memorycited by 38
  11. 2023Relational Context Learning for Human-Object Interaction Detectioncited by 36
  12. 2022Mining Cross-Person Cues for Body-Part Interactiveness Learning in HOI Detectioncited by 34
  13. 2024EZ-HOI: VLM Adaptation via Guided Prompt Learning for Zero-Shot HOI Detectioncited by 28
  14. 2023Towards Continual Egocentric Activity Recognition: A Multi-modal Egocentric Activity Dataset for Continual Learningcited by 26
  15. 2024Exploring Conditional Multi-Modal Prompts for Zero-shot HOI Detectioncited by 26
  16. 2021Exploiting Scene Graphs for Human-Object Interaction Detectioncited by 22
  17. 2022Category-Aware Transformer Network for Better Human-Object Interaction Detectioncited by 22
  18. 2023Category Query Learning for Human-Object Interaction Classificationcited by 21
  19. 2023Towards a Unified Transformer-based Framework for Scene Graph Generation and Human-object Interaction Detectioncited by 15
  20. 2025Locality-Aware Zero-Shot Human-Object Interaction Detectioncited by 12
  21. 2025HOLa: Zero-Shot HOI Detection with Low-Rank Decomposed VLM Feature Adaptationcited by 12
  22. 2024Unseen No More: Unlocking the Potential of CLIP for Generative Zero-shot HOI Detectioncited by 11
  23. 2023HODN: Disentangling Human-Object Feature for HOI Detectioncited by 9
  24. 2024A Plug-and-Play Method for Rare Human-Object Interactions Detection by Bridging Domain Gapcited by 6

Built on by

Not cited yet.