# Glance and Gaze: Inferring Action-aware Points for One-Stage Human-Object Interaction Detection

arXiv: https://arxiv.org/abs/2104.05269 · HTML: https://mlsanity.com/paper/2104.05269/

## Key facts

- Cited by 58 papers in the map (data through 14 Oct 2026).
- Builds on 28 of its 38 references that are in the map.
- First posted on arXiv 12 Apr 2021.

## Builds on (most cited first, 28 of 28)

- [Microsoft COCO: Common Objects in Context](https://mlsanity.com/paper/1405.0312/) (2014)
- [Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks](https://mlsanity.com/paper/1506.01497/) (2015)
- [Feature Pyramid Networks for Object Detection](https://mlsanity.com/paper/1612.03144/) (2016)
- [Objects as Points](https://mlsanity.com/paper/1904.07850/) (2019)
- [Stacked Hourglass Networks for Human Pose Estimation](https://mlsanity.com/paper/1603.06937/) (2016)
- [CornerNet: Detecting Objects as Paired Keypoints](https://mlsanity.com/paper/1808.01244/) (2018)
- [Deformable ConvNets v2: More Deformable, Better Results](https://mlsanity.com/paper/1811.11168/) (2018)
- [Deep Layer Aggregation](https://mlsanity.com/paper/1707.06484/) (2017)
- [Learning to Detect Human-Object Interactions](https://mlsanity.com/paper/1702.05448/) (2017)
- [Detecting and Recognizing Human-Object Interactions](https://mlsanity.com/paper/1704.07333/) (2017)
- [Visual Semantic Role Labeling](https://mlsanity.com/paper/1505.04474/) (2015)
- [Learning Human-Object Interactions by Graph Parsing Neural Networks](https://mlsanity.com/paper/1808.07962/) (2018)
- [GPS-Net: Graph Property Sensing Network for Scene Graph Generation](https://mlsanity.com/paper/2003.12962/) (2020)
- [PPDM: Parallel Point Detection and Matching for Real-time Human-Object Interaction Detection](https://mlsanity.com/paper/1912.12898/) (2019)
- [Visual Compositional Learning for Human-Object Interaction Detection](https://mlsanity.com/paper/2007.12407/) (2020)
- [iCAN: Instance-Centric Attention Network for Human-Object Interaction Detection](https://mlsanity.com/paper/1808.10437/) (2018)
- [DRG: Dual Relation Graph for Human-Object Interaction Detection](https://mlsanity.com/paper/2008.11714/) (2020)
- [VSGNet: Spatial Attention Network for Detecting Human Object Interactions Using Graph Convolutions](https://mlsanity.com/paper/2003.05541/) (2020)
- [Learning Human-Object Interaction Detection using Interaction Points](https://mlsanity.com/paper/2003.14023/) (2020)
- [Detailed 2D-3D Joint Representation for Human-Object Interaction](https://mlsanity.com/paper/2004.08154/) (2020)
- [Pose-aware Multi-level Feature Network for Human Object Interaction Detection](https://mlsanity.com/paper/1909.08453/) (2019)
- [Detecting Human-Object Interactions with Action Co-occurrence Priors](https://mlsanity.com/paper/2007.08728/) (2020)
- [Deep Contextual Attention for Human-Object Interaction Detection](https://mlsanity.com/paper/1910.07721/) (2019)
- [ConsNet: Learning Consistency Graph for Zero-Shot Human-Object Interaction Detection](https://mlsanity.com/paper/2008.06254/) (2020)
- [Cascaded Human-Object Interaction Recognition](https://mlsanity.com/paper/2003.04262/) (2020)
- [Polysemy Deciphering Network for Robust Human-Object Interaction Detection](https://mlsanity.com/paper/2008.02918/) (2020)
- [No-Frills Human-Object Interaction Detection: Factorization, Layout Encodings, and Training Techniques](https://mlsanity.com/paper/1811.05967/) (2018)
- [SaccadeNet: A Fast and Accurate Object Detector](https://mlsanity.com/paper/2003.12125/) (2020)

## Built on by (most cited first, 30 of 58)

- [GEN-VLKT: Simplify Association and Enhance Interaction Understanding for HOI Detection](https://mlsanity.com/paper/2203.13954/) (2022)
- [Mining the Benefits of Two-stage and One-stage HOI Detection](https://mlsanity.com/paper/2108.05077/) (2021)
- [RLIP: Relational Language-Image Pre-training for Human-Object Interaction Detection](https://mlsanity.com/paper/2209.01814/) (2022)
- [Exploring Structure-aware Transformer over Interaction Proposals for Human-Object Interaction Detection](https://mlsanity.com/paper/2206.06291/) (2022)
- [RLIPv2: Fast Scaling of Relational Language-Image Pre-training](https://mlsanity.com/paper/2308.09351/) (2023)
- [ViPLO: Vision Transformer based Pose-Conditioned Self-Loop Graph for Human-Object Interaction Detection](https://mlsanity.com/paper/2304.08114/) (2023)
- [Interactiveness Field in Human-Object Interactions](https://mlsanity.com/paper/2204.07718/) (2022)
- [Human-Object Interaction Detection via Disentangled Transformer](https://mlsanity.com/paper/2204.09290/) (2022)
- [Efficient Adaptive Human-Object Interaction Detection with Concept-guided Memory](https://mlsanity.com/paper/2309.03696/) (2023)
- [Relational Context Learning for Human-Object Interaction Detection](https://mlsanity.com/paper/2304.04997/) (2023)
- [What to look at and where: Semantic and Spatial Refined Transformer for detecting human-object interactions](https://mlsanity.com/paper/2204.00746/) (2022)
- [MSTR: Multi-Scale Transformer for End-to-End Human-Object Interaction Detection](https://mlsanity.com/paper/2203.14709/) (2022)
- [Mining Cross-Person Cues for Body-Part Interactiveness Learning in HOI Detection](https://mlsanity.com/paper/2207.14192/) (2022)
- [Detecting Any Human-Object Interaction Relationship: Universal HOI Detector with Spatial Prompt Learning on Foundation Models](https://mlsanity.com/paper/2311.03799/) (2023)
- [FGAHOI: Fine-Grained Anchors for Human-Object Interaction Detection](https://mlsanity.com/paper/2301.04019/) (2023)
- [QAHOI: Query-Based Anchors for Human-Object Interaction Detection](https://mlsanity.com/paper/2112.08647/) (2021)
- [Exploring the Potential of Large Foundation Models for Open-Vocabulary HOI Detection](https://mlsanity.com/paper/2404.06194/) (2024)
- [Detecting Human-Object Interactions with Object-Guided Cross-Modal Calibrated Semantics](https://mlsanity.com/paper/2202.00259/) (2022)
- [Category Query Learning for Human-Object Interaction Classification](https://mlsanity.com/paper/2303.14005/) (2023)
- [Iwin: Human-Object Interaction Detection via Transformer with Irregular Windows](https://mlsanity.com/paper/2203.10537/) (2022)
- [Learning from Observer Gaze:Zero-Shot Attention Prediction Oriented by Human-Object Interaction Recognition](https://mlsanity.com/paper/2405.09931/) (2024)
- [Discovering Human-Object Interaction Concepts via Self-Compositional Learning](https://mlsanity.com/paper/2203.14272/) (2022)
- [Unified Visual Relationship Detection with Vision and Language Models](https://mlsanity.com/paper/2303.08998/) (2023)
- [Re-mine, Learn and Reason: Exploring the Cross-modal Semantic Correlations for Language-guided HOI detection](https://mlsanity.com/paper/2307.13529/) (2023)
- [Towards Hard-Positive Query Mining for DETR-based Human-Object Interaction Detection](https://mlsanity.com/paper/2207.05293/) (2022)
- [Chairs Can be Stood on: Overcoming Object Bias in Human-Object Interaction Detection](https://mlsanity.com/paper/2207.02400/) (2022)
- [HODN: Disentangling Human-Object Feature for HOI Detection](https://mlsanity.com/paper/2308.10158/) (2023)
- [The Overlooked Classifier in Human-Object Interaction Recognition](https://mlsanity.com/paper/2112.06392/) (2021)
- [Distance Matters in Human-Object Interaction Detection](https://mlsanity.com/paper/2207.01869/) (2022)
- [Agglomerative Transformer for Human-Object Interaction Detection](https://mlsanity.com/paper/2308.08370/) (2023)

## Abstract (arXiv)

Modern human-object interaction (HOI) detection approaches can be divided into one-stage methods and twostage ones. One-stage models are more efficient due to their straightforward architectures, but the two-stage models are still advantageous in accuracy. Existing one-stage models usually begin by detecting predefined interaction areas or points, and then attend to these areas only for interaction prediction; therefore, they lack reasoning steps that dynamically search for discriminative cues. In this paper, we propose a novel one-stage method, namely Glance and Gaze Network (GGNet), which adaptively models a set of actionaware points (ActPoints) via glance and gaze steps. The glance step quickly determines whether each pixel in the feature maps is an interaction point. The gaze step leverages feature maps produced by the glance step to adaptively infer ActPoints around each pixel in a progressive manner. Features of the refined ActPoints are aggregated for interaction prediction. Moreover, we design an actionaware approach that effectively matches each detected interaction with its associated human-object pair, along with a novel hard negative attentive loss to improve the optimization of GGNet. All the above operations are conducted simultaneously and efficiently for all pixels in the feature maps. Finally, GGNet outperforms state-of-the-art methods by significant margins on both V-COCO and HICODET benchmarks. Code of GGNet is available at https: //github.com/SherlockHolmes221/GGNet.