Gaze Beyond the Frame: Forecasting Egocentric 3D Visual Span
Not on the map. Why?
Abstract (arXiv)
People continuously perceive and interact with their surroundings based on underlying intentions that drive their exploration and behaviors. While research in egocentric user and scene understanding has focused primarily on motion and contact-based interaction, forecasting human visual perception itself remains less explored despite its fundamental role in guiding human actions and its implications for AR/VR and assistive technologies. We address the challenge of egocentric 3D visual span forecasting, predicting where a person's visual perception will focus next within their three-dimensional environment. To this end, we propose EgoSpanLift, a novel method that transforms egocentric visual span forecasting from 2D image planes to 3D scenes. EgoSpanLift converts SLAM-derived keypoints into gaze-compatible geometry and extracts volumetric visual span regions. We further combine EgoSpanLift with 3D U-Net and unidirectional transformers, enabling spatio-temporal fusion to efficiently predict future visual span in the 3D grid. In addition, we curate a comprehensive benchmark from raw egocentric multisensory data, creating a testbed with 364.6K samples for 3D visual span forecasting. Our approach outperforms competitive baselines for egocentric 2D gaze anticipation and 3D localization while achieving comparable results even when projected back onto 2D image planes without additional 2D-specific training.
Key facts. Cited by 2 papers in the map (data through 14 Oct 2026). Builds on 52 of its 86 references that are in the map. First posted on arXiv 23 Nov 2025.
Builds on
52 of 86 references.
References · 52
- 2017Attention Is All You Needcited by 60,361
- 2021Ego4D: Around the World in 3,000 Hours of Egocentric Videocited by 1,556
- 20163D U-Net: Learning Dense Volumetric Segmentation from Sparse Annotationcited by 1,476
- 2015ORB-SLAM: a Versatile and Accurate Monocular SLAM Systemcited by 988
- 2018Scaling Egocentric Vision: The EPIC-KITCHENS Datasetcited by 795
- 2018Open3D: A Modern Library for 3D Data Processingcited by 721
- 2018GPyTorch: Blackbox Matrix-Matrix Gaussian Process Inference with GPU Accelerationcited by 609
- 2023Ego-Exo4D: Understanding Skilled Human Activity from First- and Third-Person Perspectivescited by 490
- 2016Direct Sparse Odometrycited by 485
- 2017VINS-Mono: A Robust and Versatile Monocular Visual-Inertial State Estimatorcited by 423
- 2019Learning joint reconstruction of hands and manipulated objectscited by 378
- 2022HOI4D: A 4D Egocentric Dataset for Category-Level Human-Object Interactioncited by 327
38 more
- 2021DexYCB: A Benchmark for Capturing Hand Grasping of Objectscited by 325
- 2023VoxFormer: Sparse Voxel Transformer for Camera-based 3D Semantic Scene Completioncited by 313
- 2023OccFormer: Dual-path Transformer for Vision-based 3D Semantic Occupancy Predictioncited by 293
- 2021H2O: Two Hands Manipulating Objects for First Person Interaction Recognitioncited by 254
- 2017First-Person Hand Action Benchmark with RGB-D Videos and 3D Hand Pose Annotationscited by 230
- 2023Project Aria: A New Tool for Egocentric Multi-Modal AI Researchcited by 208
- 2016What do different evaluation metrics tell us about saliency models?cited by 185
- 2016Eye Tracking for Everyonecited by 155
- 2015Appearance-Based Gaze Estimation in the Wildcited by 153
- 2019H+O: Unified Egocentric Recognition of 3D Hand-Object Poses and Interactionscited by 141
- 2022EPIC-KITCHENS VISOR Benchmark: VIdeo Segmentations and Object Relationscited by 140
- 2019What Would You Expect? Anticipating Egocentric Actions with Rolling-Unrolling LSTMs and Modality Attentioncited by 105
- 2022Back to MLP: A Simple Baseline for Human Motion Predictioncited by 100
- 2019Ego-Pose Estimation and Forecasting as Real-Time PD Controlcited by 87
- 2021Appearance-based Gaze Estimation With Deep Learning: A Review and Benchmarkcited by 82
- 2018Predicting Gaze in Egocentric Video by Learning Task-dependent Attention Transitioncited by 78
- 2022Fine-Grained Egocentric Hand-Object Segmentation: Dataset, Model, and Applicationscited by 78
- 2020In the Eye of the Beholder: Gaze and Actions in First Person Videocited by 69
- 2023AssemblyHands: Towards Egocentric Activity Understanding via 3D Hand Pose Estimationcited by 64
- 2023Grounding 3D Object Affordance from 2D Interactions in Imagescited by 62
- 2022In the Eye of Transformer: Global-Local Correlation for Egocentric Gaze Estimationcited by 51
- 2022End-to-End Human-Gaze-Target Detection with Transformerscited by 47
- 2024EgoLifter: Open-world 3D Segmentation for Egocentric Perceptioncited by 47
- 2021Learning to Anticipate Egocentric Actions by Imaginationcited by 40
- 2023Uncertainty-aware State Space Transformer for Egocentric 3D Hand Trajectory Forecastingcited by 34
- 2022Egocentric Prediction of Action Target in 3Dcited by 31
- 2021Egocentric Pose Estimation from Human Vision Spancited by 27
- 2024EgoChoir: Capturing 3D Human-Object Interaction Regions from Egocentric Viewscited by 26
- 2016Improving sentence compression by learning to predict gazecited by 24
- 2023Listen to Look into the Future: Audio-Visual Egocentric Gaze Anticipationcited by 20
- 2024Learning from Observer Gaze:Zero-Shot Attention Prediction Oriented by Human-Object Interaction Recognitioncited by 17
- 2024Seeing Eye to AI: Human Alignment via Gaze-Based Response Rewards for Large Language Modelscited by 12
- 2017Tracking Gaze and Visual Focus of Attention of People Involved in Social Interactioncited by 11
- 2024EgoCast: Forecasting Egocentric Human Pose in the Wildcited by 11
- 2024EgoPAT3Dv2: Predicting 3D Action Target from 2D Egocentric Vision for Human-Robot Interactioncited by 10
- 2020Ego2Hands: A Dataset for Egocentric Two-hand Segmentation and Detectioncited by 9
- 2024FIction: 4D Future Interaction Prediction from Videocited by 9
- 2018Forecasting User Attention During Everyday Mobile Interactions Using Device-Integrated and Wearable Sensorscited by 6
50 most cited of 52 All 52 (JSON)
Built on by
Citing papers per year: 2026: 2.
2 papers.