# Predicting Gaze in Egocentric Video by Learning Task-dependent Attention Transition

arXiv: https://arxiv.org/abs/1803.09125 · HTML: https://mlsanity.com/paper/1803.09125/

## Key facts

- Cited by 78 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 24 Mar 2018.

## Built on by (most cited first, 30 of 78)

- [Affordance Diffusion: Synthesizing Hand-Object Interactions](https://mlsanity.com/paper/2303.12538/) (2023)
- [Ego-Exo: Transferring Visual Representations from Third-person to First-person Videos](https://mlsanity.com/paper/2104.07905/) (2021)
- [EgoExoLearn: A Dataset for Bridging Asynchronous Ego- and Exo-centric View of Procedural Activities in Real World](https://mlsanity.com/paper/2403.16182/) (2024)
- [An Outlook into the Future of Egocentric Vision](https://mlsanity.com/paper/2308.07123/) (2023)
- [Learning Affordance Grounding from Exocentric Images](https://mlsanity.com/paper/2203.09905/) (2022)
- [My View is the Best View: Procedure Learning from Egocentric Videos](https://mlsanity.com/paper/2207.10883/) (2022)
- [Goal-Oriented Gaze Estimation for Zero-Shot Learning](https://mlsanity.com/paper/2103.03433/) (2021)
- [RoboCodeX: Multimodal Code Generation for Robotic Behavior Synthesis](https://mlsanity.com/paper/2402.16117/) (2024)
- [In the Eye of Transformer: Global-Local Correlation for Egocentric Gaze Estimation](https://mlsanity.com/paper/2208.04464/) (2022)
- [EgoVideo: Exploring Egocentric Foundation Model and Downstream Adaptation](https://mlsanity.com/paper/2406.18070/) (2024)
- [Retrieval-Augmented Egocentric Video Captioning](https://mlsanity.com/paper/2401.00789/) (2024)
- [Memory-and-Anticipation Transformer for Online Action Understanding](https://mlsanity.com/paper/2308.07893/) (2023)
- [Predicting the Future from First Person (Egocentric) Vision: A Survey](https://mlsanity.com/paper/2107.13411/) (2021)
- [Compound Prototype Matching for Few-shot Action Recognition](https://mlsanity.com/paper/2207.05515/) (2022)
- [Affordance Grounding from Demonstration Video to Target Image](https://mlsanity.com/paper/2303.14644/) (2023)
- [LEGO: Learning EGOcentric Action Frame Generation via Visual Instruction Tuning](https://mlsanity.com/paper/2312.03849/) (2023)
- [Learning Visual Affordance Grounding from Demonstration Videos](https://mlsanity.com/paper/2108.05675/) (2021)
- [LvBench: A Benchmark for Long-form Video Understanding with Versatile Multi-modal Question Answering](https://mlsanity.com/paper/2312.04817/) (2023)
- [EgoExoBench: A Benchmark for First- and Third-person View Video Understanding in MLLMs](https://mlsanity.com/paper/2507.18342/) (2025)
- [Grounded Affordance from Exocentric View](https://mlsanity.com/paper/2208.13196/) (2022)
- [In the Eye of MLLM: Benchmarking Egocentric Video Intent Understanding with Gaze-Guided Prompting](https://mlsanity.com/paper/2509.07447/) (2025)
- [ST-MTL: Spatio-Temporal Multitask Learning Model to Predict Scanpath While Tracking Instruments in Robotic Surgery](https://mlsanity.com/paper/2112.08189/) (2021)
- [Listen to Look into the Future: Audio-Visual Egocentric Gaze Anticipation](https://mlsanity.com/paper/2305.03907/) (2023)
- [Learning from Semantic Alignment between Unpaired Multiviews for Egocentric Video Recognition](https://mlsanity.com/paper/2308.11489/) (2023)
- [Modeling Fine-Grained Hand-Object Dynamics for Egocentric Video Representation Learning](https://mlsanity.com/paper/2503.00986/) (2025)
- [Vinci: A Real-time Embodied Smart Assistant based on Egocentric Vision-Language Model](https://mlsanity.com/paper/2412.21080/) (2024)
- [EgoExo-Gen: Ego-centric Video Prediction by Watching Exo-centric Videos](https://mlsanity.com/paper/2504.11732/) (2025)
- [Egocentric Auditory Attention Localization in Conversations](https://mlsanity.com/paper/2303.16024/) (2023)
- [Masked Video and Body-worn IMU Autoencoder for Egocentric Action Recognition](https://mlsanity.com/paper/2407.06628/) (2024)
- [XAIR: A Framework of Explainable AI in Augmented Reality](https://mlsanity.com/paper/2303.16292/) (2023)

## Abstract (arXiv)

We present a new computational model for gaze prediction in egocentric videos by exploring patterns in temporal shift of gaze fixations (attention transition) that are dependent on egocentric manipulation tasks. Our assumption is that the high-level context of how a task is completed in a certain way has a strong influence on attention transition and should be modeled for gaze prediction in natural dynamic scenes. Specifically, we propose a hybrid model based on deep neural networks which integrates task-dependent attention transition with bottom-up saliency prediction. In particular, the task-dependent attention transition is learned with a recurrent neural network to exploit the temporal context of gaze fixations, e.g. looking at a cup after moving gaze away from a grasped bottle. Experiments on public egocentric activity datasets show that our model significantly outperforms state-of-the-art gaze prediction methods and is able to learn meaningful transition of human attention.