# Towards Hard-Positive Query Mining for DETR-based Human-Object Interaction Detection

arXiv: https://arxiv.org/abs/2207.05293 · HTML: https://mlsanity.com/paper/2207.05293/

## Key facts

- Cited by 12 papers in the map (data through 14 Oct 2026).
- Builds on 42 of its 56 references that are in the map.
- First posted on arXiv 12 Jul 2022.

## Builds on (most cited first, 30 of 42)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Microsoft COCO: Common Objects in Context](https://mlsanity.com/paper/1405.0312/) (2014)
- [Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks](https://mlsanity.com/paper/1506.01497/) (2015)
- [End-to-End Object Detection with Transformers](https://mlsanity.com/paper/2005.12872/) (2020)
- [Deformable DETR: Deformable Transformers for End-to-End Object Detection](https://mlsanity.com/paper/2010.04159/) (2020)
- [Generalized Intersection over Union: A Metric and A Loss for Bounding Box Regression](https://mlsanity.com/paper/1902.09630/) (2019)
- [Training Region-based Object Detectors with Online Hard Example Mining](https://mlsanity.com/paper/1604.03540/) (2016)
- [DAB-DETR: Dynamic Anchor Boxes are Better Queries for DETR](https://mlsanity.com/paper/2201.12329/) (2022)
- [DN-DETR: Accelerate DETR Training by Introducing Query DeNoising](https://mlsanity.com/paper/2203.01305/) (2022)
- [Conditional DETR for Fast Training Convergence](https://mlsanity.com/paper/2108.06152/) (2021)
- [DropBlock: A regularization method for convolutional networks](https://mlsanity.com/paper/1810.12890/) (2018)
- [Action Genome: Actions as Composition of Spatio-temporal Scene Graphs](https://mlsanity.com/paper/1912.06992/) (2019)
- [Learning to Detect Human-Object Interactions](https://mlsanity.com/paper/1702.05448/) (2017)
- [Detecting and Recognizing Human-Object Interactions](https://mlsanity.com/paper/1704.07333/) (2017)
- [Visual Semantic Role Labeling](https://mlsanity.com/paper/1505.04474/) (2015)
- [HOTR: End-to-End Human-Object Interaction Detection with Transformers](https://mlsanity.com/paper/2104.13682/) (2021)
- [Fast Convergence of DETR with Spatially Modulated Co-Attention](https://mlsanity.com/paper/2101.07448/) (2021)
- [QPIC: Query-Based Pairwise Human-Object Interaction Detection with Image-Wide Contextual Information](https://mlsanity.com/paper/2103.05399/) (2021)
- [PPDM: Parallel Point Detection and Matching for Real-time Human-Object Interaction Detection](https://mlsanity.com/paper/1912.12898/) (2019)
- [End-to-End Human Object Interaction Detection with HOI Transformer](https://mlsanity.com/paper/2103.04503/) (2021)
- [Visual Compositional Learning for Human-Object Interaction Detection](https://mlsanity.com/paper/2007.12407/) (2020)
- [iCAN: Instance-Centric Attention Network for Human-Object Interaction Detection](https://mlsanity.com/paper/1808.10437/) (2018)
- [DRG: Dual Relation Graph for Human-Object Interaction Detection](https://mlsanity.com/paper/2008.11714/) (2020)
- [VSGNet: Spatial Attention Network for Detecting Human Object Interactions Using Graph Convolutions](https://mlsanity.com/paper/2003.05541/) (2020)
- [Learning Human-Object Interaction Detection using Interaction Points](https://mlsanity.com/paper/2003.14023/) (2020)
- [Mining the Benefits of Two-stage and One-stage HOI Detection](https://mlsanity.com/paper/2108.05077/) (2021)
- [Reformulating HOI Detection as Adaptive Set Prediction](https://mlsanity.com/paper/2103.05983/) (2021)
- [Detailed 2D-3D Joint Representation for Human-Object Interaction](https://mlsanity.com/paper/2004.08154/) (2020)
- [Pose-aware Multi-level Feature Network for Human Object Interaction Detection](https://mlsanity.com/paper/1909.08453/) (2019)

## Built on by (most cited first, 12 of 12)

- [HOICLIP: Efficient Knowledge Transfer for HOI Detection with Vision-Language Models](https://mlsanity.com/paper/2303.15786/) (2023)
- [Exploring Conditional Multi-Modal Prompts for Zero-shot HOI Detection](https://mlsanity.com/paper/2408.02484/) (2024)
- [Neural-Logic Human-Object Interaction Detection](https://mlsanity.com/paper/2311.09817/) (2023)
- [Category Query Learning for Human-Object Interaction Classification](https://mlsanity.com/paper/2303.14005/) (2023)
- [Human-Object Interaction Detection Collaborated with Large Relation-driven Diffusion Models](https://mlsanity.com/paper/2410.20155/) (2024)
- [Unified Visual Relationship Detection with Vision and Language Models](https://mlsanity.com/paper/2303.08998/) (2023)
- [Diagnosing Human-object Interaction Detectors](https://mlsanity.com/paper/2308.08529/) (2023)
- [Focusing on what to decode and what to train: SOV Decoding with Specific Target Guided DeNoising and Vision Language Advisor](https://mlsanity.com/paper/2307.02291/) (2023)
- [Improving Human-Object Interaction Detection via Virtual Image Learning](https://mlsanity.com/paper/2308.02606/) (2023)
- [DQEN: Dual Query Enhancement Network for DETR-based HOI Detection](https://mlsanity.com/paper/2508.18896/) (2025)
- [Disentangled Interaction Representation for One-Stage Human-Object Interaction Detection](https://mlsanity.com/paper/2312.01713/) (2023)
- [Generalized Visual Relation Detection with Diffusion Models](https://mlsanity.com/paper/2504.12100/) (2025)

## Abstract (arXiv)

Human-Object Interaction (HOI) detection is a core task for high-level image understanding. Recently, Detection Transformer (DETR)-based HOI detectors have become popular due to their superior performance and efficient structure. However, these approaches typically adopt fixed HOI queries for all testing images, which is vulnerable to the location change of objects in one specific image. Accordingly, in this paper, we propose to enhance DETR's robustness by mining hard-positive queries, which are forced to make correct predictions using partial visual cues. First, we explicitly compose hard-positive queries according to the ground-truth (GT) position of labeled human-object pairs for each training image. Specifically, we shift the GT bounding boxes of each labeled human-object pair so that the shifted boxes cover only a certain portion of the GT ones. We encode the coordinates of the shifted boxes for each labeled human-object pair into an HOI query. Second, we implicitly construct another set of hard-positive queries by masking the top scores in cross-attention maps of the decoder layers. The masked attention maps then only cover partial important cues for HOI predictions. Finally, an alternate strategy is proposed that efficiently combines both types of hard queries. In each iteration, both DETR's learnable queries and one selected type of hard-positive queries are adopted for loss computation. Experimental results show that our proposed approach can be widely applied to existing DETR-based HOI detectors. Moreover, we consistently achieve state-of-the-art performance on three benchmarks: HICO-DET, V-COCO, and HOI-A. Code is available at https://github.com/MuchHair/HQM.