# AV-Gaze: A Study on the Effectiveness of Audio Guided Visual Attention Estimation for Non-Profilic Faces

arXiv: https://arxiv.org/abs/2207.03048 · HTML: https://mlsanity.com/paper/2207.03048/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 17 of its 36 references that are in the map.
- First posted on arXiv 7 Jul 2022.

## Builds on (most cited first, 17 of 17)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Rethinking the Inception Architecture for Computer Vision](https://mlsanity.com/paper/1512.00567/) (2015)
- [SqueezeNet: AlexNet-level accuracy with 50x fewer parameters and <0.5MB model size](https://mlsanity.com/paper/1602.07360/) (2016)
- [Gaze360: Physically Unconstrained Gaze Estimation in the Wild](https://mlsanity.com/paper/1910.10088/) (2019)
- [ETH-XGaze: A Large Scale Dataset for Gaze Estimation under Extreme Head Pose and Gaze Variation](https://mlsanity.com/paper/2007.15837/) (2020)
- [MPIIGaze: Real-World Dataset and Deep Appearance-Based Gaze Estimation](https://mlsanity.com/paper/1711.09017/) (2017)
- [Few-Shot Adaptive Gaze Estimation](https://mlsanity.com/paper/1905.01941/) (2019)
- [img2pose: Face Alignment and Detection via 6DoF, Face Pose Estimation](https://mlsanity.com/paper/2012.07791/) (2020)
- [Deep Pictorial Gaze Estimation](https://mlsanity.com/paper/1807.10002/) (2018)
- [Improving Few-Shot User-Specific Gaze Adaptation via Gaze Redirection Synthesis](https://mlsanity.com/paper/1904.10638/) (2019)
- [Unsupervised Representation Learning for Gaze Estimation](https://mlsanity.com/paper/1911.06939/) (2019)
- [Automatic Gaze Analysis: A Survey of Deep Learning based Approaches](https://mlsanity.com/paper/2108.05479/) (2021)
- [Weakly-Supervised Physically Unconstrained Gaze Estimation](https://mlsanity.com/paper/2105.09803/) (2021)
- [Driver Gaze Zone Estimation using Convolutional Neural Networks: A General Framework and Ablative Analysis](https://mlsanity.com/paper/1802.02690/) (2018)
- [Speak2Label: Using Domain Knowledge for Creating a Large Scale Driver Gaze Zone Estimation Dataset](https://mlsanity.com/paper/2004.05973/) (2020)
- [MTGLS: Multi-Task Gaze Estimation with Limited Supervision](https://mlsanity.com/paper/2110.12100/) (2021)
- [Unsupervised Learning of Eye Gaze Representation from the Web](https://mlsanity.com/paper/1904.02459/) (2019)

## Built on by (most cited first, 2 of 2)

- [RAZE: Region Guided Self-Supervised Gaze Representation Learning](https://mlsanity.com/paper/2208.02485/) (2022)
- [CSGaze: Context-aware Social Gaze Prediction](https://mlsanity.com/paper/2511.05955/) (2025)

## Abstract (arXiv)

In challenging real-life conditions such as extreme head-pose, occlusions, and low-resolution images where the visual information fails to estimate visual attention/gaze direction, audio signals could provide important and complementary information. In this paper, we explore if audio-guided coarse head-pose can further enhance visual attention estimation performance for non-prolific faces. Since it is difficult to annotate audio signals for estimating the head-pose of the speaker, we use off-the-shelf state-of-the-art models to facilitate cross-modal weak-supervision. During the training phase, the framework learns complementary information from synchronized audio-visual modality. Our model can utilize any of the available modalities i.e. audio, visual or audio-visual for task-specific inference. It is interesting to note that, when AV-Gaze is tested on benchmark datasets with these specific modalities, it achieves competitive results on multiple datasets, while being highly adaptive toward challenging scenarios.