# Unsupervised Gaze Prediction in Egocentric Videos by Energy-based Surprise Modeling

arXiv: https://arxiv.org/abs/2001.11580 · HTML: https://mlsanity.com/paper/2001.11580/

## Key facts

- Cited by 5 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 30 Jan 2020.

## Built on by (most cited first, 5 of 5)

- [Predicting the Future from First Person (Egocentric) Vision: A Survey](https://mlsanity.com/paper/2107.13411/) (2021)
- [Discovering Novel Actions from Open World Egocentric Videos with Object-Grounded Visual Commonsense Reasoning](https://mlsanity.com/paper/2305.16602/) (2023)
- [Towards Active Vision for Action Localization with Reactive Control and Predictive Learning](https://mlsanity.com/paper/2111.05448/) (2021)
- [EASE: Embodied Active Event Perception via Self-Supervised Energy Minimization](https://mlsanity.com/paper/2506.17516/) (2025)
- [Unified Dynamic Scanpath Predictors Outperform Individually Trained Neural Models](https://mlsanity.com/paper/2405.02929/) (2024)

## Abstract (arXiv)

Egocentric perception has grown rapidly with the advent of immersive computing devices. Human gaze prediction is an important problem in analyzing egocentric videos and has primarily been tackled through either saliency-based modeling or highly supervised learning. We quantitatively analyze the generalization capabilities of supervised, deep learning models on the egocentric gaze prediction task on unseen, out-of-domain data. We find that their performance is highly dependent on the training data and is restricted to the domains specified in the training annotations. In this work, we tackle the problem of jointly predicting human gaze points and temporal segmentation of egocentric videos without using any training data. We introduce an unsupervised computational model that draws inspiration from cognitive psychology models of event perception. We use Grenander's pattern theory formalism to represent spatial-temporal features and model surprise as a mechanism to predict gaze fixation points. Extensive evaluation on two publicly available datasets - GTEA and GTEA+ datasets-shows that the proposed model can significantly outperform all unsupervised baselines and some supervised gaze prediction baselines. Finally, we show that the model can also temporally segment egocentric videos with a performance comparable to more complex, fully supervised deep learning baselines.