# A Modular Multimodal Architecture for Gaze Target Prediction: Application to Privacy-Sensitive Settings

arXiv: https://arxiv.org/abs/2307.05158 · HTML: https://mlsanity.com/paper/2307.05158/

## Key facts

- Cited by 31 papers in the map (data through 14 Oct 2026).
- Builds on 16 of its 49 references that are in the map.
- First posted on arXiv 11 Jul 2023.

## Builds on (most cited first, 16 of 16)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [ImageNet Large Scale Visual Recognition Challenge](https://mlsanity.com/paper/1409.0575/) (2014)
- [EfficientNet: Rethinking Model Scaling for Convolutional Neural Networks](https://mlsanity.com/paper/1905.11946/) (2019)
- [Feature Pyramid Networks for Object Detection](https://mlsanity.com/paper/1612.03144/) (2016)
- [Towards Robust Monocular Depth Estimation: Mixing Datasets for Zero-shot Cross-dataset Transfer](https://mlsanity.com/paper/1907.01341/) (2019)
- [Deep High-Resolution Representation Learning for Visual Recognition](https://mlsanity.com/paper/1908.07919/) (2019)
- [Gaze360: Physically Unconstrained Gaze Estimation in the Wild](https://mlsanity.com/paper/1910.10088/) (2019)
- [DeepPrivacy: A Generative Adversarial Network for Face Anonymization](https://mlsanity.com/paper/1909.04538/) (2019)
- [Detecting Attended Visual Targets in Video](https://mlsanity.com/paper/2003.02501/) (2020)
- [Connecting Gaze, Scene, and Attention: Generalized Attention Estimation via Joint Modeling of Gaze and Scene Saliency](https://mlsanity.com/paper/1807.10437/) (2018)
- [Believe It or Not, We Know What You Are Looking at!](https://mlsanity.com/paper/1907.02364/) (2019)
- [LAEO-Net: revisiting people Looking At Each Other in videos](https://mlsanity.com/paper/1906.05261/) (2019)
- [GOO: A Dataset for Gaze Object Prediction in Retail Environments](https://mlsanity.com/paper/2105.10793/) (2021)
- [Tracking Gaze and Visual Focus of Attention of People Involved in Social Interaction](https://mlsanity.com/paper/1703.04727/) (2017)
- [Do Pedestrians Pay Attention? Eye Contact Detection in the Wild](https://mlsanity.com/paper/2112.04212/) (2021)

## Built on by (most cited first, 30 of 31)

- [Gaze-LLE: Gaze Target Estimation via Large-Scale Learned Encoders](https://mlsanity.com/paper/2412.09586/) (2024)
- [Object-aware Gaze Target Detection](https://mlsanity.com/paper/2307.09662/) (2023)
- [ViTGaze: Gaze Following with Interaction Features in Vision Transformers](https://mlsanity.com/paper/2403.12778/) (2024)
- [Exploring the Zero-Shot Capabilities of Vision-Language Models for Improving Gaze Following](https://mlsanity.com/paper/2406.03907/) (2024)
- [GazeVLM: A Vision-Language Model for Multi-Task Gaze Understanding](https://mlsanity.com/paper/2511.06348/) (2025)
- [Diffusion-Refined VQA Annotations for Semi-Supervised Gaze Following](https://mlsanity.com/paper/2406.02774/) (2024)
- [Multi-view Gaze Target Estimation](https://mlsanity.com/paper/2508.05857/) (2025)
- [A Novel Framework for Multi-Person Temporal Gaze Following and Social Gaze Prediction](https://mlsanity.com/paper/2403.10511/) (2024)
- [GA3CE: Unconstrained 3D Gaze Estimation with Gaze-Aware 3D Context Encoding](https://mlsanity.com/paper/2505.10671/) (2025)
- [VL4Gaze: Unleashing Vision-Language Models for Gaze Following](https://mlsanity.com/paper/2512.20735/) (2025)
- [Sharingan: A Transformer-based Architecture for Gaze Following](https://mlsanity.com/paper/2310.00816/) (2023)
- [Upper-Body Pose-based Gaze Estimation for Privacy-Preserving 3D Gaze Target Detection](https://mlsanity.com/paper/2409.17886/) (2024)
- [GazeDETR: Gaze Detection using Disentangled Head and Gaze Representations](https://mlsanity.com/paper/2508.12966/) (2025)
- [GaTector+: A Unified Head-free Framework for Gaze Object and Gaze Following Prediction](https://mlsanity.com/paper/2510.25301/) (2025)
- [GRASP: Learning to Ground Social Reasoning in Multi-Person Non-Verbal Interactions](https://mlsanity.com/paper/2605.15764/) (2026)
- [CAPE: A CLIP-Aware Pointing Ensemble of Complementary Heatmap Cues for Embodied Reference Understanding](https://mlsanity.com/paper/2507.21888/) (2025)
- [CSGaze: Context-aware Social Gaze Prediction](https://mlsanity.com/paper/2511.05955/) (2025)
- [GazeProphetV2: Head-Movement-Based Gaze Prediction Enabling Efficient Foveated Rendering on Mobile VR](https://mlsanity.com/paper/2511.19988/) (2025)
- [GazeMoE: Perception of Gaze Target with Mixture-of-Experts](https://mlsanity.com/paper/2603.06256/) (2026)
- [End-to-End Shared Attention Estimation via Group Detection with Feedback Refinement](https://mlsanity.com/paper/2604.01714/) (2026)
- [Eyes on VLM: Benchmarking Gaze Following and Social Gaze Prediction in Vision Language Models](https://mlsanity.com/paper/2605.19859/) (2026)
- [Enhancing Gaze Reasoning in Vision Foundation Models for Gaze Following](https://mlsanity.com/paper/2605.22607/) (2026)
- [OmniGF: A Dual-Branch Vision-Language Framework for Unified Gaze Following](https://mlsanity.com/paper/2605.26399/) (2026)
- [Multi-scale Object-Aware Gaze Estimation via Geometric Reasoning](https://mlsanity.com/paper/2606.29334/) (2026)
- [SAGE: Synchronized Action-Gaze Recognition and Anticipation for Human Behavior Understanding](https://mlsanity.com/paper/2607.04017/) (2026)
- [PAGE: Towards Practical Human-level Gaze Target Estimation](https://mlsanity.com/paper/2607.04860/) (2026)
- [TextGaze: Prompting Gaze Target Estimation with Textual Scene Cues](https://mlsanity.com/paper/2607.10130/) (2026)
- [Gaze-Anchored Social Net: Decoding Implicit Relations via Joint Modeling](https://mlsanity.com/paper/2607.22847/) (2026)
- [Gaze Target Estimation Anywhere with Concepts](https://mlsanity.com/paper/2608.11367/) (2026)
- [From Gaze to Meaning: A Training-Free AI Agent for Unified Grounding and Explanation](https://mlsanity.com/paper/2609.06208/) (2026)

## Abstract (arXiv)

Predicting where a person is looking is a complex task, requiring to understand not only the person's gaze and scene content, but also the 3D scene structure and the person's situation (are they manipulating? interacting or observing others? attentive?) to detect obstructions in the line of sight or apply attention priors that humans typically have when observing others. In this paper, we hypothesize that identifying and leveraging such priors can be better achieved through the exploitation of explicitly derived multimodal cues such as depth and pose. We thus propose a modular multimodal architecture allowing to combine these cues using an attention mechanism. The architecture can naturally be exploited in privacy-sensitive situations such as surveillance and health, where personally identifiable information cannot be released. We perform extensive experiments on the GazeFollow and VideoAttentionTarget public datasets, obtaining state-of-the-art performance and demonstrating very competitive results in the privacy setting case.