# Detecting Human-Object Interactions with Object-Guided Cross-Modal Calibrated Semantics

arXiv: https://arxiv.org/abs/2202.00259 · HTML: https://mlsanity.com/paper/2202.00259/

## Key facts

- Cited by 22 papers in the map (data through 14 Oct 2026).
- Builds on 46 of its 66 references that are in the map.
- First posted on arXiv 1 Feb 2022.

## Builds on (most cited first, 30 of 46)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Microsoft COCO: Common Objects in Context](https://mlsanity.com/paper/1405.0312/) (2014)
- [End-to-End Object Detection with Transformers](https://mlsanity.com/paper/2005.12872/) (2020)
- [Layer Normalization](https://mlsanity.com/paper/1607.06450/) (2016)
- [Squeeze-and-Excitation Networks](https://mlsanity.com/paper/1709.01507/) (2017)
- [Distributed Representations of Words and Phrases and their Compositionality](https://mlsanity.com/paper/1310.4546/) (2013)
- [Unsupervised Feature Learning via Non-Parametric Instance-level Discrimination](https://mlsanity.com/paper/1805.01978/) (2018)
- [Generalized Intersection over Union: A Metric and A Loss for Bounding Box Regression](https://mlsanity.com/paper/1902.09630/) (2019)
- [Convolutional Sequence to Sequence Learning](https://mlsanity.com/paper/1705.03122/) (2017)
- [Unbiased Scene Graph Generation from Biased Training](https://mlsanity.com/paper/2002.11949/) (2020)
- [Learning to Detect Human-Object Interactions](https://mlsanity.com/paper/1702.05448/) (2017)
- [Exploring Visual Relationship for Image Captioning](https://mlsanity.com/paper/1809.07041/) (2018)
- [Detecting and Recognizing Human-Object Interactions](https://mlsanity.com/paper/1704.07333/) (2017)
- [Visual Semantic Role Labeling](https://mlsanity.com/paper/1505.04474/) (2015)
- [Zero-Shot Object Detection](https://mlsanity.com/paper/1804.04340/) (2018)
- [HOTR: End-to-End Human-Object Interaction Detection with Transformers](https://mlsanity.com/paper/2104.13682/) (2021)
- [Learning Human-Object Interactions by Graph Parsing Neural Networks](https://mlsanity.com/paper/1808.07962/) (2018)
- [GPS-Net: Graph Property Sensing Network for Scene Graph Generation](https://mlsanity.com/paper/2003.12962/) (2020)
- [Knowledge-Embedded Routing Network for Scene Graph Generation](https://mlsanity.com/paper/1903.03326/) (2019)
- [Zero-shot Recognition via Semantic Embeddings and Knowledge Graphs](https://mlsanity.com/paper/1803.08035/) (2018)
- [QPIC: Query-Based Pairwise Human-Object Interaction Detection with Image-Wide Contextual Information](https://mlsanity.com/paper/2103.05399/) (2021)
- [PPDM: Parallel Point Detection and Matching for Real-time Human-Object Interaction Detection](https://mlsanity.com/paper/1912.12898/) (2019)
- [End-to-End Human Object Interaction Detection with HOI Transformer](https://mlsanity.com/paper/2103.04503/) (2021)
- [iCAN: Instance-Centric Attention Network for Human-Object Interaction Detection](https://mlsanity.com/paper/1808.10437/) (2018)
- [DRG: Dual Relation Graph for Human-Object Interaction Detection](https://mlsanity.com/paper/2008.11714/) (2020)
- [VSGNet: Spatial Attention Network for Detecting Human Object Interactions Using Graph Convolutions](https://mlsanity.com/paper/2003.05541/) (2020)
- [Dynamic Fusion with Intra- and Inter- Modality Attention Flow for Visual Question Answering](https://mlsanity.com/paper/1812.05252/) (2018)
- [CAMP: Cross-Modal Adaptive Message Passing for Text-Image Retrieval](https://mlsanity.com/paper/1909.05506/) (2019)
- [Learning Human-Object Interaction Detection using Interaction Points](https://mlsanity.com/paper/2003.14023/) (2020)

## Built on by (most cited first, 22 of 22)

- [Full-Body Articulated Human-Object Interaction](https://mlsanity.com/paper/2212.10621/) (2022)
- [RLIP: Relational Language-Image Pre-training for Human-Object Interaction Detection](https://mlsanity.com/paper/2209.01814/) (2022)
- [RLIPv2: Fast Scaling of Relational Language-Image Pre-training](https://mlsanity.com/paper/2308.09351/) (2023)
- [Detecting Any Human-Object Interaction Relationship: Universal HOI Detector with Spatial Prompt Learning on Foundation Models](https://mlsanity.com/paper/2311.03799/) (2023)
- [F-HOI: Toward Fine-grained Semantic-Aligned 3D Human-Object Interactions](https://mlsanity.com/paper/2407.12435/) (2024)
- [Human-centric Scene Understanding for 3D Large-scale Scenarios](https://mlsanity.com/paper/2307.14392/) (2023)
- [Surgical-VQLA++: Adversarial Contrastive Learning for Calibrated Robust Visual Question-Localized Answering in Robotic Surgery](https://mlsanity.com/paper/2408.04958/) (2024)
- [Re-mine, Learn and Reason: Exploring the Cross-modal Semantic Correlations for Language-guided HOI detection](https://mlsanity.com/paper/2307.13529/) (2023)
- [Towards Hard-Positive Query Mining for DETR-based Human-Object Interaction Detection](https://mlsanity.com/paper/2207.05293/) (2022)
- [HODN: Disentangling Human-Object Feature for HOI Detection](https://mlsanity.com/paper/2308.10158/) (2023)
- [Distance Matters in Human-Object Interaction Detection](https://mlsanity.com/paper/2207.01869/) (2022)
- [Exploring Self- and Cross-Triplet Correlations for Human-Object Interaction Detection](https://mlsanity.com/paper/2401.05676/) (2024)
- [Focusing on what to decode and what to train: SOV Decoding with Specific Target Guided DeNoising and Vision Language Advisor](https://mlsanity.com/paper/2307.02291/) (2023)
- [Improving Human-Object Interaction Detection via Virtual Image Learning](https://mlsanity.com/paper/2308.02606/) (2023)
- [UAHOI: Uncertainty-aware Robust Interaction Learning for HOI Detection](https://mlsanity.com/paper/2408.07430/) (2024)
- [PhysLab: A Benchmark Dataset for Multi-Granularity Visual Parsing of Physics Experiments](https://mlsanity.com/paper/2506.06631/) (2025)
- [Enhancing HOI Detection with Contextual Cues from Large Vision-Language Models](https://mlsanity.com/paper/2311.16475/) (2023)
- [Egocentric Human-Object Interaction Detection: A New Benchmark and Method](https://mlsanity.com/paper/2506.14189/) (2025)
- [DQEN: Dual Query Enhancement Network for DETR-based HOI Detection](https://mlsanity.com/paper/2508.18896/) (2025)
- [UV R-CNN: Stable and Efficient Dense Human Pose Estimation](https://mlsanity.com/paper/2211.02337/) (2022)
- [Mining Conditional Part Semantics with Occluded Extrapolation for Human-Object Interaction Detection](https://mlsanity.com/paper/2307.10499/) (2023)
- [Exploring Interactive Semantic Alignment for Efficient HOI Detection with Vision-language Model](https://mlsanity.com/paper/2404.12678/) (2024)

## Abstract (arXiv)

Human-Object Interaction (HOI) detection is an essential task to understand human-centric images from a fine-grained perspective. Although end-to-end HOI detection models thrive, their paradigm of parallel human/object detection and verb class prediction loses two-stage methods' merit: object-guided hierarchy. The object in one HOI triplet gives direct clues to the verb to be predicted. In this paper, we aim to boost end-to-end models with object-guided statistical priors. Specifically, We propose to utilize a Verb Semantic Model (VSM) and use semantic aggregation to profit from this object-guided hierarchy. Similarity KL (SKL) loss is proposed to optimize VSM to align with the HOI dataset's priors. To overcome the static semantic embedding problem, we propose to generate cross-modality-aware visual and semantic features by Cross-Modal Calibration (CMC). The above modules combined composes Object-guided Cross-modal Calibration Network (OCN). Experiments conducted on two popular HOI detection benchmarks demonstrate the significance of incorporating the statistical prior knowledge and produce state-of-the-art performances. More detailed analysis indicates proposed modules serve as a stronger verb predictor and a more superior method of utilizing prior knowledge. The codes are available at urlhttps://github.com/JacobYuan7/OCN-HOI-Benchmark.