# Robust Egocentric Visual Attention Prediction Through Language-guided Scene Context-aware Learning

arXiv: https://arxiv.org/abs/2601.01818 · HTML: https://mlsanity.com/paper/2601.01818/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 27 of its 40 references that are in the map.
- First posted on arXiv 5 Jan 2026.

## Builds on (most cited first, 27 of 27)

- [SlowFast Networks for Video Recognition](https://mlsanity.com/paper/1812.03982/) (2018)
- [Ego4D: Around the World in 3,000 Hours of Egocentric Video](https://mlsanity.com/paper/2110.07058/) (2021)
- [Multiscale Vision Transformers](https://mlsanity.com/paper/2104.11227/) (2021)
- [VideoChat: Chat-Centric Video Understanding](https://mlsanity.com/paper/2305.06355/) (2023)
- [NV-Embed: Improved Techniques for Training LLMs as Generalist Embedding Models](https://mlsanity.com/paper/2405.17428/) (2024)
- [Assembly101: A Large-Scale Multi-View Video Dataset for Understanding Procedural Activities](https://mlsanity.com/paper/2203.14712/) (2022)
- [Egocentric Video-Language Pretraining](https://mlsanity.com/paper/2206.01670/) (2022)
- [The EPIC-KITCHENS Dataset: Collection, Challenges and Baselines](https://mlsanity.com/paper/2005.00343/) (2020)
- [Ego-Exo: Transferring Visual Representations from Third-person to First-person Videos](https://mlsanity.com/paper/2104.07905/) (2021)
- [EgoExoLearn: A Dataset for Bridging Asynchronous Ego- and Exo-centric View of Procedural Activities in Real World](https://mlsanity.com/paper/2403.16182/) (2024)
- [Predicting Gaze in Egocentric Video by Learning Task-dependent Attention Transition](https://mlsanity.com/paper/1803.09125/) (2018)
- [In the Eye of the Beholder: Gaze and Actions in First Person Video](https://mlsanity.com/paper/2006.00626/) (2020)
- [Mutual Context Network for Jointly Estimating Egocentric Gaze and Actions](https://mlsanity.com/paper/1901.01874/) (2019)
- [In the Eye of Transformer: Global-Local Correlation for Egocentric Gaze Estimation](https://mlsanity.com/paper/2208.04464/) (2022)
- [With a Little Help from my Temporal Context: Multimodal Egocentric Action Recognition](https://mlsanity.com/paper/2111.01024/) (2021)
- [Aria Everyday Activities Dataset](https://mlsanity.com/paper/2402.13349/) (2024)
- [Scene-aware Egocentric 3D Human Pose Estimation](https://mlsanity.com/paper/2212.11684/) (2022)
- [LEGO: Learning EGOcentric Action Frame Generation via Visual Instruction Tuning](https://mlsanity.com/paper/2312.03849/) (2023)
- [EgoEnv: Human-centric environment representations from egocentric video](https://mlsanity.com/paper/2207.11365/) (2022)
- [Eliciting In-Context Learning in Vision-Language Models for Videos Through Curated Data Distributional Properties](https://mlsanity.com/paper/2311.17041/) (2023)
- [Listen to Look into the Future: Audio-Visual Egocentric Gaze Anticipation](https://mlsanity.com/paper/2305.03907/) (2023)
- [Shaping embodied agent behavior with activity-context priors from egocentric video](https://mlsanity.com/paper/2110.07692/) (2021)
- [Leveraging Next-Active Objects for Context-Aware Anticipation in Egocentric Videos](https://mlsanity.com/paper/2308.08303/) (2023)
- [Digging Deeper into Egocentric Gaze Prediction](https://mlsanity.com/paper/1904.06090/) (2019)
- [A Backpack Full of Skills: Egocentric Video Understanding with Diverse Task Perspectives](https://mlsanity.com/paper/2403.03037/) (2024)
- [EgoNav: Egocentric Scene-aware Human Trajectory Prediction](https://mlsanity.com/paper/2403.19026/) (2024)
- [Egocentric Scene Understanding via Multimodal Spatial Rectifier](https://mlsanity.com/paper/2207.07077/) (2022)

## Abstract (arXiv)

As the demand for analyzing egocentric videos grows, egocentric visual attention prediction, anticipating where a camera wearer will attend, has garnered increasing attention. However, it remains challenging due to the inherent complexity and ambiguity of dynamic egocentric scenes. Motivated by evidence that scene contextual information plays a crucial role in modulating human attention, in this paper, we present a language-guided scene context-aware learning framework for robust egocentric visual attention prediction. We first design a context perceiver which is guided to summarize the egocentric video based on a language-based scene description, generating context-aware video representations. We then introduce two training objectives that: 1) encourage the framework to focus on the target point-of-interest regions and 2) suppress distractions from irrelevant regions which are less likely to attract first-person attention. Extensive experiments on Ego4D and Aria Everyday Activities (AEA) datasets demonstrate the effectiveness of our approach, achieving state-of-the-art performance and enhanced robustness across diverse, dynamic egocentric scenarios.