# Believe It or Not, We Know What You Are Looking at!

arXiv: https://arxiv.org/abs/1907.02364 · HTML: https://mlsanity.com/paper/1907.02364/

## Key facts

- Cited by 38 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 4 Jul 2019.

## Built on by (most cited first, 30 of 38)

- [End-to-End Human-Gaze-Target Detection with Transformers](https://mlsanity.com/paper/2203.10433/) (2022)
- [Gaze-LLE: Gaze Target Estimation via Large-Scale Learned Encoders](https://mlsanity.com/paper/2412.09586/) (2024)
- [ChildPlay: A New Benchmark for Understanding Children's Gaze Behaviour](https://mlsanity.com/paper/2307.01630/) (2023)
- [A Modular Multimodal Architecture for Gaze Target Prediction: Application to Privacy-Sensitive Settings](https://mlsanity.com/paper/2307.05158/) (2023)
- [Object-aware Gaze Target Detection](https://mlsanity.com/paper/2307.09662/) (2023)
- [GOO: A Dataset for Gaze Object Prediction in Retail Environments](https://mlsanity.com/paper/2105.10793/) (2021)
- [Multimodal Across Domains Gaze Target Detection](https://mlsanity.com/paper/2208.10822/) (2022)
- [Patch-level Gaze Distribution Prediction for Gaze Following](https://mlsanity.com/paper/2211.11062/) (2022)
- [GaTector: A Unified Framework for Gaze Object Prediction](https://mlsanity.com/paper/2112.03549/) (2021)
- [ViTGaze: Gaze Following with Interaction Features in Vision Transformers](https://mlsanity.com/paper/2403.12778/) (2024)
- [Exploring the Zero-Shot Capabilities of Vision-Language Models for Improving Gaze Following](https://mlsanity.com/paper/2406.03907/) (2024)
- [Towards Online Multi-Modal Social Interaction Understanding](https://mlsanity.com/paper/2503.19851/) (2025)
- [Interaction-aware Joint Attention Estimation Using People Attributes](https://mlsanity.com/paper/2308.05382/) (2023)
- [Joint Gaze-Location and Gaze-Object Detection](https://mlsanity.com/paper/2308.13857/) (2023)
- [TransGOP: Transformer-Based Gaze Object Prediction](https://mlsanity.com/paper/2402.13578/) (2024)
- [GazeHTA: End-to-end Gaze Target Detection with Head-Target Association](https://mlsanity.com/paper/2404.10718/) (2024)
- [Diffusion-Refined VQA Annotations for Semi-Supervised Gaze Following](https://mlsanity.com/paper/2406.02774/) (2024)
- [Multi-view Gaze Target Estimation](https://mlsanity.com/paper/2508.05857/) (2025)
- [MGTR: End-to-End Mutual Gaze Detection with Transformer](https://mlsanity.com/paper/2209.10930/) (2022)
- [A Novel Framework for Multi-Person Temporal Gaze Following and Social Gaze Prediction](https://mlsanity.com/paper/2403.10511/) (2024)
- [GA3CE: Unconstrained 3D Gaze Estimation with Gaze-Aware 3D Context Encoding](https://mlsanity.com/paper/2505.10671/) (2025)
- [GEM: Context-Aware Gaze EstiMation with Visual Search Behavior Matching for Chest Radiograph](https://mlsanity.com/paper/2408.05502/) (2024)
- [Sharingan: A Transformer-based Architecture for Gaze Following](https://mlsanity.com/paper/2310.00816/) (2023)
- [Driver Attention Tracking and Analysis](https://mlsanity.com/paper/2404.07122/) (2024)
- [Upper-Body Pose-based Gaze Estimation for Privacy-Preserving 3D Gaze Target Detection](https://mlsanity.com/paper/2409.17886/) (2024)
- [GazeDETR: Gaze Detection using Disentangled Head and Gaze Representations](https://mlsanity.com/paper/2508.12966/) (2025)
- [GaTector+: A Unified Head-free Framework for Gaze Object and Gaze Following Prediction](https://mlsanity.com/paper/2510.25301/) (2025)
- [GazeD: Context-Aware Diffusion for Accurate 3D Gaze Estimation](https://mlsanity.com/paper/2601.12948/) (2026)
- [GazeMoE: Perception of Gaze Target with Mixture-of-Experts](https://mlsanity.com/paper/2603.06256/) (2026)
- [Eyes on VLM: Benchmarking Gaze Following and Social Gaze Prediction in Vision Language Models](https://mlsanity.com/paper/2605.19859/) (2026)

## Abstract (arXiv)

By borrowing the wisdom of human in gaze following, we propose a two-stage solution for gaze point prediction of the target persons in a scene. Specifically, in the first stage, both head image and its position are fed into a gaze direction pathway to predict the gaze direction, and then multi-scale gaze direction fields are generated to characterize the distribution of gaze points without considering the scene contents. In the second stage, the multi-scale gaze direction fields are concatenated with the image contents and fed into a heatmap pathway for heatmap regression. There are two merits for our two-stage solution based gaze following: i) our solution mimics the behavior of human in gaze following, therefore it is more psychological plausible; ii) besides using heatmap to supervise the output of our network, we can also leverage gaze direction to facilitate the training of gaze direction pathway, therefore our network can be more robustly trained. Considering that existing gaze following dataset is annotated by the third-view persons, we build a video gaze following dataset, where the ground truth is annotated by the observers in the videos. Therefore it is more reliable. The evaluation with such a dataset reflects the capacity of different methods in real scenarios better. Extensive experiments on both datasets show that our method significantly outperforms existing methods, which validates the effectiveness of our solution for gaze following. Our dataset and codes are released in https://github.com/svip-lab/GazeFollowing.