mlsanityData to 7 Oct 2026

Detecting Human-Object Interactions with Object-Guided Cross-Modal Calibrated Semantics

arXiv 2202.00259 ↗First posted 1 Feb 2022cs.CVcited by 22

Not on the map. Why?

Abstract (arXiv)

Human-Object Interaction (HOI) detection is an essential task to understand human-centric images from a fine-grained perspective. Although end-to-end HOI detection models thrive, their paradigm of parallel human/object detection and verb class prediction loses two-stage methods' merit: object-guided hierarchy. The object in one HOI triplet gives direct clues to the verb to be predicted. In this paper, we aim to boost end-to-end models with object-guided statistical priors. Specifically, We propose to utilize a Verb Semantic Model (VSM) and use semantic aggregation to profit from this object-guided hierarchy. Similarity KL (SKL) loss is proposed to optimize VSM to align with the HOI dataset's priors. To overcome the static semantic embedding problem, we propose to generate cross-modality-aware visual and semantic features by Cross-Modal Calibration (CMC). The above modules combined composes Object-guided Cross-modal Calibration Network (OCN). Experiments conducted on two popular HOI detection benchmarks demonstrate the significance of incorporating the statistical prior knowledge and produce state-of-the-art performances. More detailed analysis indicates proposed modules serve as a stronger verb predictor and a more superior method of utilizing prior knowledge. The codes are available at urlhttps://github.com/JacobYuan7/OCN-HOI-Benchmark.

Key facts. Cited by 22 papers in the map (data through 7 Oct 2026). Builds on 46 of its 66 references that are in the map. First posted on arXiv 1 Feb 2022.

Builds on

46 of 66 references.

References · 46

  1. 2017Attention Is All You Needcited by 59,932
  2. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  3. 2014Microsoft COCO: Common Objects in Contextcited by 17,498
  4. 2020End-to-End Object Detection with Transformerscited by 7,384
  5. 2016Layer Normalizationcited by 5,579
  6. 2017Squeeze-and-Excitation Networkscited by 4,763
  7. 2013Distributed Representations of Words and Phrases and their Compositionalitycited by 3,462
  8. 2018Unsupervised Feature Learning via Non-Parametric Instance-level Discriminationcited by 1,838
  9. 2019Generalized Intersection over Union: A Metric and A Loss for Bounding Box Regressioncited by 1,361
  10. 2017Convolutional Sequence to Sequence Learningcited by 550
  11. 2020Unbiased Scene Graph Generation from Biased Trainingcited by 355
  12. 2017Learning to Detect Human-Object Interactionscited by 238
34 more
  1. 2018Exploring Visual Relationship for Image Captioningcited by 215
  2. 2017Detecting and Recognizing Human-Object Interactionscited by 199
  3. 2018Zero-Shot Object Detectioncited by 189
  4. 2015Visual Semantic Role Labelingcited by 188
  5. 2021HOTR: End-to-End Human-Object Interaction Detection with Transformerscited by 183
  6. 2018Learning Human-Object Interactions by Graph Parsing Neural Networkscited by 176
  7. 2020GPS-Net: Graph Property Sensing Network for Scene Graph Generationcited by 165
  8. 2019Knowledge-Embedded Routing Network for Scene Graph Generationcited by 163
  9. 2018Zero-shot Recognition via Semantic Embeddings and Knowledge Graphscited by 155
  10. 2021QPIC: Query-Based Pairwise Human-Object Interaction Detection with Image-Wide Contextual Informationcited by 147
  11. 2019PPDM: Parallel Point Detection and Matching for Real-time Human-Object Interaction Detectioncited by 143
  12. 2021End-to-End Human Object Interaction Detection with HOI Transformercited by 134
  13. 2018iCAN: Instance-Centric Attention Network for Human-Object Interaction Detectioncited by 126
  14. 2020DRG: Dual Relation Graph for Human-Object Interaction Detectioncited by 117
  15. 2020VSGNet: Spatial Attention Network for Detecting Human Object Interactions Using Graph Convolutionscited by 115
  16. 2018Dynamic Fusion with Intra- and Inter- Modality Attention Flow for Visual Question Answeringcited by 113
  17. 2019CAMP: Cross-Modal Adaptive Message Passing for Text-Image Retrievalcited by 110
  18. 2020Learning Human-Object Interaction Detection using Interaction Pointscited by 108
  19. 2021Reformulating HOI Detection as Adaptive Set Predictioncited by 95
  20. 2019Improving Referring Expression Grounding with Cross-modal Attention-guided Erasingcited by 88
  21. 2019Pose-aware Multi-level Feature Network for Human Object Interaction Detectioncited by 88
  22. 2020HOI Analysis: Integrating and Decomposing Human-Object Interactioncited by 81
  23. 2021Semantic Relation Reasoning for Shot-Stable Few-Shot Object Detectioncited by 79
  24. 2019Cross-Modality Attention with Semantic Graph Embedding for Multi-Label Classificationcited by 78
  25. 2020Detecting Human-Object Interactions with Action Co-occurrence Priorscited by 72
  26. 2021Glance and Gaze: Inferring Action-aware Points for One-Stage Human-Object Interaction Detectioncited by 58
  27. 2019Deep Contextual Attention for Human-Object Interaction Detectioncited by 54
  28. 2019Detecting Human-Object Interactions via Functional Generalizationcited by 53
  29. 2020Contextual Heterogeneous Graph Network for Human-Object Interaction Detectioncited by 50
  30. 2021Spatio-Temporal Dynamic Inference Network for Group Activity Recognitioncited by 42
  31. 2018Learning to Count Objects in Natural Images for Visual Question Answeringcited by 40
  32. 2020DIRV: Dense Interaction Region Voting for End-to-End Human-Object Interaction Detectioncited by 31
  33. 2021Visualizing Data using GTSNEcited by 23
  34. 2020Any-Shot Object Detectioncited by 10

Built on by

Cited by year

Citing papers per year: 2022: 5; 2023: 9; 2024: 5; 2025: 3.

22 papers.

Sort:

By area:

  1. 2022Full-Body Articulated Human-Object Interactioncited by 76
  2. 2022RLIP: Relational Language-Image Pre-training for Human-Object Interaction Detectioncited by 54
  3. 2023RLIPv2: Fast Scaling of Relational Language-Image Pre-trainingcited by 44
  4. 2023Detecting Any Human-Object Interaction Relationship: Universal HOI Detector with Spatial Prompt Learning on Foundation Modelscited by 34
  5. 2024F-HOI: Toward Fine-grained Semantic-Aligned 3D Human-Object Interactionscited by 24
  6. 2023Human-centric Scene Understanding for 3D Large-scale Scenarioscited by 17
  7. 2024Surgical-VQLA++: Adversarial Contrastive Learning for Calibrated Robust Visual Question-Localized Answering in Robotic Surgerycited by 17
  8. 2023Re-mine, Learn and Reason: Exploring the Cross-modal Semantic Correlations for Language-guided HOI detectioncited by 13
  9. 2022Towards Hard-Positive Query Mining for DETR-based Human-Object Interaction Detectioncited by 12
  10. 2023HODN: Disentangling Human-Object Feature for HOI Detectioncited by 9
  11. 2022Distance Matters in Human-Object Interaction Detectioncited by 8
  12. 2024Exploring Self- and Cross-Triplet Correlations for Human-Object Interaction Detectioncited by 7
10 more
  1. 2023Focusing on what to decode and what to train: SOV Decoding with Specific Target Guided DeNoising and Vision Language Advisorcited by 4
  2. 2023Improving Human-Object Interaction Detection via Virtual Image Learningcited by 4
  3. 2024UAHOI: Uncertainty-aware Robust Interaction Learning for HOI Detectioncited by 4
  4. 2023Enhancing HOI Detection with Contextual Cues from Large Vision-Language Modelscited by 3
  5. 2025PhysLab: A Benchmark Dataset for Multi-Granularity Visual Parsing of Physics Experimentscited by 3
  6. 2025Egocentric Human-Object Interaction Detection: A New Benchmark and Methodcited by 3
  7. 2025DQEN: Dual Query Enhancement Network for DETR-based HOI Detectioncited by 1
  8. 2022UV R-CNN: Stable and Efficient Dense Human Pose Estimationcited by 0
  9. 2023Mining Conditional Part Semantics with Occluded Extrapolation for Human-Object Interaction Detectioncited by 0
  10. 2024Exploring Interactive Semantic Alignment for Efficient HOI Detection with Vision-language Modelcited by 0