# Connecting Gaze, Scene, and Attention: Generalized Attention Estimation via Joint Modeling of Gaze and Scene Saliency

arXiv: https://arxiv.org/abs/1807.10437 · HTML: https://mlsanity.com/paper/1807.10437/

## Key facts

- Cited by 53 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 27 Jul 2018.

## Built on by (most cited first, 30 of 53)

- [Appearance-based Gaze Estimation With Deep Learning: A Review and Benchmark](https://mlsanity.com/paper/2104.12668/) (2021)
- [In the Eye of Transformer: Global-Local Correlation for Egocentric Gaze Estimation](https://mlsanity.com/paper/2208.04464/) (2022)
- [End-to-End Human-Gaze-Target Detection with Transformers](https://mlsanity.com/paper/2203.10433/) (2022)
- [Gaze-LLE: Gaze Target Estimation via Large-Scale Learned Encoders](https://mlsanity.com/paper/2412.09586/) (2024)
- [ChildPlay: A New Benchmark for Understanding Children's Gaze Behaviour](https://mlsanity.com/paper/2307.01630/) (2023)
- [Automatic Gaze Analysis: A Survey of Deep Learning based Approaches](https://mlsanity.com/paper/2108.05479/) (2021)
- [A Modular Multimodal Architecture for Gaze Target Prediction: Application to Privacy-Sensitive Settings](https://mlsanity.com/paper/2307.05158/) (2023)
- [Weakly-Supervised Physically Unconstrained Gaze Estimation](https://mlsanity.com/paper/2105.09803/) (2021)
- [Object-aware Gaze Target Detection](https://mlsanity.com/paper/2307.09662/) (2023)
- [GOO: A Dataset for Gaze Object Prediction in Retail Environments](https://mlsanity.com/paper/2105.10793/) (2021)
- [Multimodal Across Domains Gaze Target Detection](https://mlsanity.com/paper/2208.10822/) (2022)
- [Patch-level Gaze Distribution Prediction for Gaze Following](https://mlsanity.com/paper/2211.11062/) (2022)
- [XAIR: A Framework of Explainable AI in Augmented Reality](https://mlsanity.com/paper/2303.16292/) (2023)
- [Understanding Embodied Reference with Touch-Line Transformer](https://mlsanity.com/paper/2210.05668/) (2022)
- [DVGaze: Dual-View Gaze Estimation](https://mlsanity.com/paper/2308.10310/) (2023)
- [ViTGaze: Gaze Following with Interaction Features in Vision Transformers](https://mlsanity.com/paper/2403.12778/) (2024)
- [Spatio-Temporal Attention and Gaussian Processes for Personalized Video Gaze Estimation](https://mlsanity.com/paper/2404.05215/) (2024)
- [Enhancing 3D Gaze Estimation in the Wild using Weak Supervision with Gaze Following Labels](https://mlsanity.com/paper/2502.20249/) (2025)
- [GazeVLM: A Vision-Language Model for Multi-Task Gaze Understanding](https://mlsanity.com/paper/2511.06348/) (2025)
- [A Gaze-grounded Visual Question Answering Dataset for Clarifying Ambiguous Japanese Questions](https://mlsanity.com/paper/2403.17545/) (2024)
- [Interaction-aware Joint Attention Estimation Using People Attributes](https://mlsanity.com/paper/2308.05382/) (2023)
- [Joint Gaze-Location and Gaze-Object Detection](https://mlsanity.com/paper/2308.13857/) (2023)
- [TransGOP: Transformer-Based Gaze Object Prediction](https://mlsanity.com/paper/2402.13578/) (2024)
- [GazeHTA: End-to-end Gaze Target Detection with Head-Target Association](https://mlsanity.com/paper/2404.10718/) (2024)
- [The Story in Your Eyes: An Individual-difference-aware Model for Cross-person Gaze Estimation](https://mlsanity.com/paper/2106.14183/) (2021)
- [Diffusion-Refined VQA Annotations for Semi-Supervised Gaze Following](https://mlsanity.com/paper/2406.02774/) (2024)
- [Multi-view Gaze Target Estimation](https://mlsanity.com/paper/2508.05857/) (2025)
- [GEM: Context-Aware Gaze EstiMation with Visual Search Behavior Matching for Chest Radiograph](https://mlsanity.com/paper/2408.05502/) (2024)
- [Leveraging Multi-Modal Saliency and Fusion for Gaze Target Detection](https://mlsanity.com/paper/2504.19271/) (2025)
- [CASPER: Cognitive Architecture for Social Perception and Engagement in Robots](https://mlsanity.com/paper/2209.01012/) (2022)

## Abstract (arXiv)

This paper addresses the challenging problem of estimating the general visual attention of people in images. Our proposed method is designed to work across multiple naturalistic social scenarios and provides a full picture of the subject's attention and gaze. In contrast, earlier works on gaze and attention estimation have focused on constrained problems in more specific contexts. In particular, our model explicitly represents the gaze direction and handles out-of-frame gaze targets. We leverage three different datasets using a multi-task learning approach. We evaluate our method on widely used benchmarks for single-tasks such as gaze angle estimation and attention-within-an-image, as well as on the new challenging task of generalized visual attention prediction. In addition, we have created extended annotations for the MMDB and GazeFollow datasets which are used in our experiments, which we will publicly release.