# Mutual Context Network for Jointly Estimating Egocentric Gaze and Actions

arXiv: https://arxiv.org/abs/1901.01874 · HTML: https://mlsanity.com/paper/1901.01874/

## Key facts

- Cited by 54 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 7 Jan 2019.

## Built on by (most cited first, 30 of 54)

- [Video Mamba Suite: State Space Model as a Versatile Alternative for Video Understanding](https://mlsanity.com/paper/2403.09626/) (2024)
- [EgoExoLearn: A Dataset for Bridging Asynchronous Ego- and Exo-centric View of Procedural Activities in Real World](https://mlsanity.com/paper/2403.16182/) (2024)
- [Goal-Oriented Gaze Estimation for Zero-Shot Learning](https://mlsanity.com/paper/2103.03433/) (2021)
- [In the Eye of Transformer: Global-Local Correlation for Egocentric Gaze Estimation](https://mlsanity.com/paper/2208.04464/) (2022)
- [With a Little Help from my Temporal Context: Multimodal Egocentric Action Recognition](https://mlsanity.com/paper/2111.01024/) (2021)
- [Memory-and-Anticipation Transformer for Online Action Understanding](https://mlsanity.com/paper/2308.07893/) (2023)
- [Predicting the Future from First Person (Egocentric) Vision: A Survey](https://mlsanity.com/paper/2107.13411/) (2021)
- [LEGO: Learning EGOcentric Action Frame Generation via Visual Instruction Tuning](https://mlsanity.com/paper/2312.03849/) (2023)
- [EgoExoBench: A Benchmark for First- and Third-person View Video Understanding in MLLMs](https://mlsanity.com/paper/2507.18342/) (2025)
- [In the Eye of MLLM: Benchmarking Egocentric Video Intent Understanding with Gaze-Guided Prompting](https://mlsanity.com/paper/2509.07447/) (2025)
- [Listen to Look into the Future: Audio-Visual Egocentric Gaze Anticipation](https://mlsanity.com/paper/2305.03907/) (2023)
- [Modeling Fine-Grained Hand-Object Dynamics for Egocentric Video Representation Learning](https://mlsanity.com/paper/2503.00986/) (2025)
- [Challenges and Trends in Egocentric Vision: A Survey](https://mlsanity.com/paper/2503.15275/) (2025)
- [RadioTransformer: A Cascaded Global-Focal Transformer for Visual Attention-guided Disease Classification](https://mlsanity.com/paper/2202.11781/) (2022)
- [Egocentric Video Task Translation](https://mlsanity.com/paper/2212.06301/) (2022)
- [Vinci: A Real-time Embodied Smart Assistant based on Egocentric Vision-Language Model](https://mlsanity.com/paper/2412.21080/) (2024)
- [Egocentric Auditory Attention Localization in Conversations](https://mlsanity.com/paper/2303.16024/) (2023)
- [Masked Video and Body-worn IMU Autoencoder for Egocentric Action Recognition](https://mlsanity.com/paper/2407.06628/) (2024)
- [EAGLE: Egocentric AGgregated Language-video Engine](https://mlsanity.com/paper/2409.17523/) (2024)
- [Fine-grained Affordance Annotation for Egocentric Hand-Object Interaction Videos](https://mlsanity.com/paper/2302.03292/) (2023)
- [Recent Advances in Leveraging Human Guidance for Sequential Decision-Making Tasks](https://mlsanity.com/paper/2107.05825/) (2021)
- [Gaze-VLM:Bridging Gaze and VLMs through Attention Regularization for Egocentric Understanding](https://mlsanity.com/paper/2510.21356/) (2025)
- [UniLS: End-to-End Audio-Driven Avatars for Unified Listening and Speaking](https://mlsanity.com/paper/2512.09327/) (2025)
- [Towards Interactive Intelligence for Digital Humans](https://mlsanity.com/paper/2512.13674/) (2025)
- [A Backpack Full of Skills: Egocentric Video Understanding with Diverse Task Perspectives](https://mlsanity.com/paper/2403.03037/) (2024)
- [Gaze-Regularized Vision-Language-Action Models for Robotic Manipulation](https://mlsanity.com/paper/2603.23202/) (2026)
- [SOS! Self-supervised Learning Over Sets Of Handled Objects In Egocentric Action Recognition](https://mlsanity.com/paper/2204.04796/) (2022)
- [GPT4Ego: Unleashing the Potential of Pre-trained Models for Zero-Shot Egocentric Action Recognition](https://mlsanity.com/paper/2401.10039/) (2024)
- [An Egocentric Vision-Language Model based Portable Real-time Smart Assistant](https://mlsanity.com/paper/2503.04250/) (2025)
- [Bridging Perspectives: A Survey on Cross-view Collaborative Intelligence with Egocentric-Exocentric Vision](https://mlsanity.com/paper/2506.06253/) (2025)

## Abstract (arXiv)

In this work, we address two coupled tasks of gaze prediction and action recognition in egocentric videos by exploring their mutual context. Our assumption is that in the procedure of performing a manipulation task, what a person is doing determines where the person is looking at, and the gaze point reveals gaze and non-gaze regions which contain important and complementary information about the undergoing action. We propose a novel mutual context network (MCN) that jointly learns action-dependent gaze prediction and gaze-guided action recognition in an end-to-end manner. Experiments on public egocentric video datasets demonstrate that our MCN achieves state-of-the-art performance of both gaze prediction and action recognition.