# GaTector: A Unified Framework for Gaze Object Prediction

arXiv: https://arxiv.org/abs/2112.03549 · HTML: https://mlsanity.com/paper/2112.03549/

## Key facts

- Cited by 18 papers in the map (data through 14 Oct 2026).
- Builds on 26 of its 66 references that are in the map.
- First posted on arXiv 7 Dec 2021.

## Builds on (most cited first, 26 of 26)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [PyTorch: An Imperative Style, High-Performance Deep Learning Library](https://mlsanity.com/paper/1912.01703/) (2019)
- [Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks](https://mlsanity.com/paper/1506.01497/) (2015)
- [Mask R-CNN](https://mlsanity.com/paper/1703.06870/) (2017)
- [You Only Look Once: Unified, Real-Time Object Detection](https://mlsanity.com/paper/1506.02640/) (2015)
- [SSD: Single Shot MultiBox Detector](https://mlsanity.com/paper/1512.02325/) (2015)
- [YOLOv3: An Incremental Improvement](https://mlsanity.com/paper/1804.02767/) (2018)
- [YOLOv4: Optimal Speed and Accuracy of Object Detection](https://mlsanity.com/paper/2004.10934/) (2020)
- [FCOS: Fully Convolutional One-Stage Object Detection](https://mlsanity.com/paper/1904.01355/) (2019)
- [YOLO9000: Better, Faster, Stronger](https://mlsanity.com/paper/1612.08242/) (2016)
- [Focal Loss for Dense Object Detection](https://mlsanity.com/paper/1708.02002/) (2017)
- [Distance-IoU Loss: Faster and Better Learning for Bounding Box Regression](https://mlsanity.com/paper/1911.08287/) (2019)
- [Probabilistic Anchor Assignment with IoU Prediction for Object Detection](https://mlsanity.com/paper/2007.08103/) (2020)
- [Detecting Attended Visual Targets in Video](https://mlsanity.com/paper/2003.02501/) (2020)
- [Goal-Oriented Gaze Estimation for Zero-Shot Learning](https://mlsanity.com/paper/2103.03433/) (2021)
- [Motion-Attentive Transition for Zero-Shot Video Object Segmentation](https://mlsanity.com/paper/2003.04253/) (2020)
- [Generalizing Gaze Estimation with Outlier-guided Collaborative Adaptation](https://mlsanity.com/paper/2107.13780/) (2021)
- [Cascaded Human-Object Interaction Recognition](https://mlsanity.com/paper/2003.04262/) (2020)
- [Believe It or Not, We Know What You Are Looking at!](https://mlsanity.com/paper/1907.02364/) (2019)
- [Side-Aware Boundary Localization for More Precise Object Detection](https://mlsanity.com/paper/1912.04260/) (2019)
- [RPC: A Large-Scale Retail Product Checkout Dataset](https://mlsanity.com/paper/1901.07249/) (2019)
- [GOO: A Dataset for Gaze Object Prediction in Retail Environments](https://mlsanity.com/paper/2105.10793/) (2021)
- [Background-Click Supervision for Temporal Action Localization](https://mlsanity.com/paper/2111.12449/) (2021)
- [MVTec D2S: Densely Segmented Supermarket Dataset](https://mlsanity.com/paper/1804.08292/) (2018)
- [Extended Gaze Following: Detecting Objects in Videos Beyond the Camera Field of View](https://mlsanity.com/paper/1902.10953/) (2019)

## Built on by (most cited first, 18 of 18)

- [Gaze-LLE: Gaze Target Estimation via Large-Scale Learned Encoders](https://mlsanity.com/paper/2412.09586/) (2024)
- [ChildPlay: A New Benchmark for Understanding Children's Gaze Behaviour](https://mlsanity.com/paper/2307.01630/) (2023)
- [Patch-level Gaze Distribution Prediction for Gaze Following](https://mlsanity.com/paper/2211.11062/) (2022)
- [Robust Region Feature Synthesizer for Zero-Shot Object Detection](https://mlsanity.com/paper/2201.00103/) (2022)
- [ViTGaze: Gaze Following with Interaction Features in Vision Transformers](https://mlsanity.com/paper/2403.12778/) (2024)
- [GazeVLM: A Vision-Language Model for Multi-Task Gaze Understanding](https://mlsanity.com/paper/2511.06348/) (2025)
- [Fewer is More: Efficient Object Detection in Large Aerial Images](https://mlsanity.com/paper/2212.13136/) (2022)
- [Joint Gaze-Location and Gaze-Object Detection](https://mlsanity.com/paper/2308.13857/) (2023)
- [TransGOP: Transformer-Based Gaze Object Prediction](https://mlsanity.com/paper/2402.13578/) (2024)
- [Multi-level Graph Subspace Contrastive Learning for Hyperspectral Image Clustering](https://mlsanity.com/paper/2404.05211/) (2024)
- [GA3CE: Unconstrained 3D Gaze Estimation with Gaze-Aware 3D Context Encoding](https://mlsanity.com/paper/2505.10671/) (2025)
- [GEM: Context-Aware Gaze EstiMation with Visual Search Behavior Matching for Chest Radiograph](https://mlsanity.com/paper/2408.05502/) (2024)
- [Learning Video-independent Eye Contact Segmentation from In-the-Wild Videos](https://mlsanity.com/paper/2210.02033/) (2022)
- [GaTector+: A Unified Head-free Framework for Gaze Object and Gaze Following Prediction](https://mlsanity.com/paper/2510.25301/) (2025)
- [S2RC-GCN: A Spatial-Spectral Reliable Contrastive Graph Convolutional Network for Complex Land Cover Classification Using Hyperspectral Images](https://mlsanity.com/paper/2404.00964/) (2024)
- [DHECA-SuperGaze: Dual Head-Eye Cross-Attention and Super-Resolution for Unconstrained Gaze Estimation](https://mlsanity.com/paper/2505.08426/) (2025)
- [Multi-scale Object-Aware Gaze Estimation via Geometric Reasoning](https://mlsanity.com/paper/2606.29334/) (2026)
- [Open-Vocabulary Gaze Object Prediction: Benchmark and Method](https://mlsanity.com/paper/2607.18827/) (2026)

## Abstract (arXiv)

Gaze object prediction is a newly proposed task that aims to discover the objects being stared at by humans. It is of great application significance but still lacks a unified solution framework. An intuitive solution is to incorporate an object detection branch into an existing gaze prediction method. However, previous gaze prediction methods usually use two different networks to extract features from scene image and head image, which would lead to heavy network architecture and prevent each branch from joint optimization. In this paper, we build a novel framework named GaTector to tackle the gaze object prediction problem in a unified way. Particularly, a specific-general-specific (SGS) feature extractor is firstly proposed to utilize a shared backbone to extract general features for both scene and head images. To better consider the specificity of inputs and tasks, SGS introduces two input-specific blocks before the shared backbone and three task-specific blocks after the shared backbone. Specifically, a novel Defocus layer is designed to generate object-specific features for the object detection task without losing information or requiring extra computations. Moreover, the energy aggregation loss is introduced to guide the gaze heatmap to concentrate on the stared box. In the end, we propose a novel wUoC metric that can reveal the difference between boxes even when they share no overlapping area. Extensive experiments on the GOO dataset verify the superiority of our method in all three tracks, i.e. object detection, gaze estimation, and gaze object prediction.