mlsanityData to 14 Oct 2026

Gaze Beyond the Frame: Forecasting Egocentric 3D Visual Span

arXiv 2511.18470 ↗First posted 23 Nov 2025cs.CVcited by 2

Not on the map. Why?

Abstract (arXiv)

People continuously perceive and interact with their surroundings based on underlying intentions that drive their exploration and behaviors. While research in egocentric user and scene understanding has focused primarily on motion and contact-based interaction, forecasting human visual perception itself remains less explored despite its fundamental role in guiding human actions and its implications for AR/VR and assistive technologies. We address the challenge of egocentric 3D visual span forecasting, predicting where a person's visual perception will focus next within their three-dimensional environment. To this end, we propose EgoSpanLift, a novel method that transforms egocentric visual span forecasting from 2D image planes to 3D scenes. EgoSpanLift converts SLAM-derived keypoints into gaze-compatible geometry and extracts volumetric visual span regions. We further combine EgoSpanLift with 3D U-Net and unidirectional transformers, enabling spatio-temporal fusion to efficiently predict future visual span in the 3D grid. In addition, we curate a comprehensive benchmark from raw egocentric multisensory data, creating a testbed with 364.6K samples for 3D visual span forecasting. Our approach outperforms competitive baselines for egocentric 2D gaze anticipation and 3D localization while achieving comparable results even when projected back onto 2D image planes without additional 2D-specific training.

Key facts. Cited by 2 papers in the map (data through 14 Oct 2026). Builds on 52 of its 86 references that are in the map. First posted on arXiv 23 Nov 2025.

Builds on

52 of 86 references.

References · 52

  1. 2017Attention Is All You Needcited by 60,361
  2. 2021Ego4D: Around the World in 3,000 Hours of Egocentric Videocited by 1,556
  3. 20163D U-Net: Learning Dense Volumetric Segmentation from Sparse Annotationcited by 1,476
  4. 2015ORB-SLAM: a Versatile and Accurate Monocular SLAM Systemcited by 988
  5. 2018Scaling Egocentric Vision: The EPIC-KITCHENS Datasetcited by 795
  6. 2018Open3D: A Modern Library for 3D Data Processingcited by 721
  7. 2018GPyTorch: Blackbox Matrix-Matrix Gaussian Process Inference with GPU Accelerationcited by 609
  8. 2023Ego-Exo4D: Understanding Skilled Human Activity from First- and Third-Person Perspectivescited by 490
  9. 2016Direct Sparse Odometrycited by 485
  10. 2017VINS-Mono: A Robust and Versatile Monocular Visual-Inertial State Estimatorcited by 423
  11. 2019Learning joint reconstruction of hands and manipulated objectscited by 378
  12. 2022HOI4D: A 4D Egocentric Dataset for Category-Level Human-Object Interactioncited by 327
38 more
  1. 2021DexYCB: A Benchmark for Capturing Hand Grasping of Objectscited by 325
  2. 2023VoxFormer: Sparse Voxel Transformer for Camera-based 3D Semantic Scene Completioncited by 313
  3. 2023OccFormer: Dual-path Transformer for Vision-based 3D Semantic Occupancy Predictioncited by 293
  4. 2021H2O: Two Hands Manipulating Objects for First Person Interaction Recognitioncited by 254
  5. 2017First-Person Hand Action Benchmark with RGB-D Videos and 3D Hand Pose Annotationscited by 230
  6. 2023Project Aria: A New Tool for Egocentric Multi-Modal AI Researchcited by 208
  7. 2016What do different evaluation metrics tell us about saliency models?cited by 185
  8. 2016Eye Tracking for Everyonecited by 155
  9. 2015Appearance-Based Gaze Estimation in the Wildcited by 153
  10. 2019H+O: Unified Egocentric Recognition of 3D Hand-Object Poses and Interactionscited by 141
  11. 2022EPIC-KITCHENS VISOR Benchmark: VIdeo Segmentations and Object Relationscited by 140
  12. 2019What Would You Expect? Anticipating Egocentric Actions with Rolling-Unrolling LSTMs and Modality Attentioncited by 105
  13. 2022Back to MLP: A Simple Baseline for Human Motion Predictioncited by 100
  14. 2019Ego-Pose Estimation and Forecasting as Real-Time PD Controlcited by 87
  15. 2021Appearance-based Gaze Estimation With Deep Learning: A Review and Benchmarkcited by 82
  16. 2018Predicting Gaze in Egocentric Video by Learning Task-dependent Attention Transitioncited by 78
  17. 2022Fine-Grained Egocentric Hand-Object Segmentation: Dataset, Model, and Applicationscited by 78
  18. 2020In the Eye of the Beholder: Gaze and Actions in First Person Videocited by 69
  19. 2023AssemblyHands: Towards Egocentric Activity Understanding via 3D Hand Pose Estimationcited by 64
  20. 2023Grounding 3D Object Affordance from 2D Interactions in Imagescited by 62
  21. 2022In the Eye of Transformer: Global-Local Correlation for Egocentric Gaze Estimationcited by 51
  22. 2022End-to-End Human-Gaze-Target Detection with Transformerscited by 47
  23. 2024EgoLifter: Open-world 3D Segmentation for Egocentric Perceptioncited by 47
  24. 2021Learning to Anticipate Egocentric Actions by Imaginationcited by 40
  25. 2023Uncertainty-aware State Space Transformer for Egocentric 3D Hand Trajectory Forecastingcited by 34
  26. 2022Egocentric Prediction of Action Target in 3Dcited by 31
  27. 2021Egocentric Pose Estimation from Human Vision Spancited by 27
  28. 2024EgoChoir: Capturing 3D Human-Object Interaction Regions from Egocentric Viewscited by 26
  29. 2016Improving sentence compression by learning to predict gazecited by 24
  30. 2023Listen to Look into the Future: Audio-Visual Egocentric Gaze Anticipationcited by 20
  31. 2024Learning from Observer Gaze:Zero-Shot Attention Prediction Oriented by Human-Object Interaction Recognitioncited by 17
  32. 2024Seeing Eye to AI: Human Alignment via Gaze-Based Response Rewards for Large Language Modelscited by 12
  33. 2017Tracking Gaze and Visual Focus of Attention of People Involved in Social Interactioncited by 11
  34. 2024EgoCast: Forecasting Egocentric Human Pose in the Wildcited by 11
  35. 2024EgoPAT3Dv2: Predicting 3D Action Target from 2D Egocentric Vision for Human-Robot Interactioncited by 10
  36. 2020Ego2Hands: A Dataset for Egocentric Two-hand Segmentation and Detectioncited by 9
  37. 2024FIction: 4D Future Interaction Prediction from Videocited by 9
  38. 2018Forecasting User Attention During Everyday Mobile Interactions Using Device-Integrated and Wearable Sensorscited by 6

50 most cited of 52 All 52 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 2.

2 papers.