# Learning User Embeddings from Human Gaze for Personalised Saliency Prediction

arXiv: https://arxiv.org/abs/2403.13653 · HTML: https://mlsanity.com/paper/2403.13653/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- Builds on 20 of its 68 references that are in the map.
- First posted on arXiv 20 Mar 2024.

## Builds on (most cited first, 20 of 20)

- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift](https://mlsanity.com/paper/1502.03167/) (2015)
- [In Defense of the Triplet Loss for Person Re-Identification](https://mlsanity.com/paper/1703.07737/) (2017)
- [Network In Network](https://mlsanity.com/paper/1312.4400/) (2013)
- [Deeply-Supervised Nets](https://mlsanity.com/paper/1409.5185/) (2014)
- [Salient Object Detection: A Survey](https://mlsanity.com/paper/1411.5878/) (2014)
- [DeepGaze IIE: Calibrated prediction in and out-of-domain for state-of-the-art saliency modeling](https://mlsanity.com/paper/2105.12441/) (2021)
- [A Deep Multi-Level Network for Saliency Prediction](https://mlsanity.com/paper/1609.01064/) (2016)
- [Predicting Goal-directed Human Attention Using Inverse Reinforcement Learning](https://mlsanity.com/paper/2005.14310/) (2020)
- [Shallow and Deep Convolutional Networks for Saliency Prediction](https://mlsanity.com/paper/1603.00845/) (2016)
- [Gazeformer: Scalable, Effective and Fast Prediction of Goal-Directed Human Attention](https://mlsanity.com/paper/2303.15274/) (2023)
- [UserIdentifier: Implicit User Representations for Simple and Effective Personalized Sentiment Analysis](https://mlsanity.com/paper/2110.00135/) (2021)
- [Multimodal Integration of Human-Like Attention in Visual Question Answering](https://mlsanity.com/paper/2109.13139/) (2021)
- [VQA-MHUG: A Gaze Dataset to Study Multimodal Neural Attention in Visual Question Answering](https://mlsanity.com/paper/2109.13116/) (2021)
- [Personalized Saliency and its Prediction](https://mlsanity.com/paper/1710.03011/) (2017)
- [Author2Vec: A Framework for Generating User Embedding](https://mlsanity.com/paper/2003.11627/) (2020)
- [Neural Photofit: Gaze-based Mental Image Reconstruction](https://mlsanity.com/paper/2108.07524/) (2021)
- [Gaze-enhanced Crossmodal Embeddings for Emotion Recognition](https://mlsanity.com/paper/2205.00129/) (2022)
- [Personalization of Saliency Estimation](https://mlsanity.com/paper/1711.08000/) (2017)
- [Gaze Distribution Analysis and Saliency Prediction Across Age Groups](https://mlsanity.com/paper/1705.07284/) (2017)

## Built on by (most cited first, 4 of 4)

- [Reading Recognition in the Wild](https://mlsanity.com/paper/2505.24848/) (2025)
- [Few-shot Personalized Scanpath Prediction](https://mlsanity.com/paper/2504.05499/) (2025)
- [PRE-MAP: Personalized Reinforced Eye-tracking Multimodal LLM for High-Resolution Multi-Attribute Point Prediction](https://mlsanity.com/paper/2507.19213/) (2025)
- [Through Their Eyes: Fixation-aligned Tuning for Personalized User Emulation](https://mlsanity.com/paper/2604.09368/) (2026)

## Abstract (arXiv)

Reusable embeddings of user behaviour have shown significant performance improvements for the personalised saliency prediction task. However, prior works require explicit user characteristics and preferences as input, which are often difficult to obtain. We present a novel method to extract user embeddings from pairs of natural images and corresponding saliency maps generated from a small amount of user-specific eye tracking data. At the core of our method is a Siamese convolutional neural encoder that learns the user embeddings by contrasting the image and personal saliency map pairs of different users. Evaluations on two public saliency datasets show that the generated embeddings have high discriminative power, are effective at refining universal saliency maps to the individual users, and generalise well across users and images. Finally, based on our model's ability to encode individual user characteristics, our work points towards other applications that can benefit from reusable embeddings of gaze behaviour.