# Gaze-enhanced Crossmodal Embeddings for Emotion Recognition

arXiv: https://arxiv.org/abs/2205.00129 · HTML: https://mlsanity.com/paper/2205.00129/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- Builds on 13 of its 68 references that are in the map.
- First posted on arXiv 30 Apr 2022.

## Builds on (most cited first, 13 of 13)

- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Appearance-Based Gaze Estimation in the Wild](https://mlsanity.com/paper/1504.02863/) (2015)
- [Rendering of Eyes for Eye-Shape Registration and Gaze Estimation](https://mlsanity.com/paper/1505.05916/) (2015)
- [Pupil: An Open Source Platform for Pervasive Eye Tracking and Mobile Gaze-based Interaction](https://mlsanity.com/paper/1405.0006/) (2014)
- [Leveraging Recent Advances in Deep Learning for Audio-Visual Emotion Recognition](https://mlsanity.com/paper/2103.09154/) (2021)
- [The OMG-Emotion Behavior Dataset](https://mlsanity.com/paper/1803.05434/) (2018)
- [Detecting Low Rapport During Natural Interactions in Small Groups from Non-Verbal Behaviour](https://mlsanity.com/paper/1801.06055/) (2018)
- [Deep Learning for Human Affect Recognition: Insights and New Developments](https://mlsanity.com/paper/1901.02884/) (2019)
- [EmoBed: Strengthening Monomodal Emotion Recognition via Training with Crossmodal Emotion Embeddings](https://mlsanity.com/paper/1907.10428/) (2019)
- [Robust Latent Representations via Cross-Modal Translation and Alignment](https://mlsanity.com/paper/2011.01631/) (2020)
- [Emergent Leadership Detection Across Datasets](https://mlsanity.com/paper/1905.02058/) (2019)
- [Affective computing using speech and eye gaze: a review and bimodal system proposal for continuous affect prediction](https://mlsanity.com/paper/1805.06652/) (2018)
- [Eye-based Continuous Affect Prediction](https://mlsanity.com/paper/1907.09896/) (2019)

## Built on by (most cited first, 4 of 4)

- [Learning User Embeddings from Human Gaze for Personalised Saliency Prediction](https://mlsanity.com/paper/2403.13653/) (2024)
- [Exploring Emotion Expression Recognition in Older Adults Interacting with a Virtual Coach](https://mlsanity.com/paper/2311.05567/) (2023)
- [QualitEye: Public and Privacy-preserving Gaze Data Quality Verification](https://mlsanity.com/paper/2506.05908/) (2025)
- [An Intelligent Decision Support System for Emotion Monitoring using Microscopic Fixational Dynamics](https://mlsanity.com/paper/2609.00846/) (2026)

## Abstract (arXiv)

Emotional expressions are inherently multimodal -- integrating facial behavior, speech, and gaze -- but their automatic recognition is often limited to a single modality, e.g. speech during a phone call. While previous work proposed crossmodal emotion embeddings to improve monomodal recognition performance, despite its importance, an explicit representation of gaze was not included. We propose a new approach to emotion recognition that incorporates an explicit representation of gaze in a crossmodal emotion embedding framework. We show that our method outperforms the previous state of the art for both audio-only and video-only emotion classification on the popular One-Minute Gradual Emotion Recognition dataset. Furthermore, we report extensive ablation experiments and provide detailed insights into the performance of different state-of-the-art gaze representations and integration strategies. Our results not only underline the importance of gaze for emotion recognition but also demonstrate a practical and highly effective approach to leveraging gaze information for this task.