# An Outlook into the Future of Egocentric Vision

arXiv: https://arxiv.org/abs/2308.07123 · HTML: https://mlsanity.com/paper/2308.07123/

## Key facts

- Cited by 84 papers in the map (data through 14 Oct 2026).
- Builds on 201 of its 401 references that are in the map.
- First posted on arXiv 14 Aug 2023.

## Builds on (most cited first, 30 of 201)

- [ImageNet Large Scale Visual Recognition Challenge](https://mlsanity.com/paper/1409.0575/) (2014)
- [ViViT: A Video Vision Transformer](https://mlsanity.com/paper/2103.15691/) (2021)
- [Ego4D: Around the World in 3,000 Hours of Egocentric Video](https://mlsanity.com/paper/2110.07058/) (2021)
- [LSUN: Construction of a Large-scale Image Dataset using Deep Learning with Humans in the Loop](https://mlsanity.com/paper/1506.03365/) (2015)
- [ImageBind: One Embedding Space To Bind Them All](https://mlsanity.com/paper/2305.05665/) (2023)
- [Vision-and-Language Navigation: Interpreting visually-grounded navigation instructions in real environments](https://mlsanity.com/paper/1711.07280/) (2017)
- [NetVLAD: CNN architecture for weakly supervised place recognition](https://mlsanity.com/paper/1511.07247/) (2015)
- [VizWiz Grand Challenge: Answering Visual Questions from Blind People](https://mlsanity.com/paper/1802.08218/) (2018)
- [Scaling Egocentric Vision: The EPIC-KITCHENS Dataset](https://mlsanity.com/paper/1804.02748/) (2018)
- [R3M: A Universal Visual Representation for Robot Manipulation](https://mlsanity.com/paper/2203.12601/) (2022)
- [UNISURF: Unifying Neural Implicit Surfaces and Radiance Fields for Multi-View Reconstruction](https://mlsanity.com/paper/2104.10078/) (2021)
- [Rescaling Egocentric Vision](https://mlsanity.com/paper/2006.13256/) (2020)
- [VIP: Towards Universal Visual Reward and Representation via Value-Implicit Pre-Training](https://mlsanity.com/paper/2210.00030/) (2022)
- [Learning joint reconstruction of hands and manipulated objects](https://mlsanity.com/paper/1904.05767/) (2019)
- [Embodied Question Answering](https://mlsanity.com/paper/1711.11543/) (2017)
- [HOI4D: A 4D Egocentric Dataset for Category-Level Human-Object Interaction](https://mlsanity.com/paper/2203.01577/) (2022)
- [SQA3D: Situated Question Answering in 3D Scenes](https://mlsanity.com/paper/2210.07474/) (2022)
- [ARCTIC: A Dataset for Dexterous Bimanual Hand-Object Manipulation](https://mlsanity.com/paper/2204.13662/) (2022)
- [Assembly101: A Large-Scale Multi-View Video Dataset for Understanding Procedural Activities](https://mlsanity.com/paper/2203.14712/) (2022)
- [Understanding Human Hands in Contact at Internet Scale](https://mlsanity.com/paper/2006.06669/) (2020)
- [Real-World Robot Learning with Masked Visual Pre-training](https://mlsanity.com/paper/2210.03109/) (2022)
- [First-Person Hand Action Benchmark with RGB-D Videos and 3D Hand Pose Annotations](https://mlsanity.com/paper/1704.02463/) (2017)
- [EPIC-Fusion: Audio-Visual Temporal Binding for Egocentric Action Recognition](https://mlsanity.com/paper/1908.08498/) (2019)
- [Keeping Your Eye on the Ball: Trajectory Attention in Video Transformers](https://mlsanity.com/paper/2106.05392/) (2021)
- [Omnivore: A Single Model for Many Visual Modalities](https://mlsanity.com/paper/2201.08377/) (2022)
- [Learning Video Representations from Large Language Models](https://mlsanity.com/paper/2212.04501/) (2022)
- [Anticipative Video Transformer](https://mlsanity.com/paper/2106.02036/) (2021)
- [MeMViT: Memory-Augmented Multiscale Vision Transformer for Efficient Long-Term Video Recognition](https://mlsanity.com/paper/2201.08383/) (2022)
- [Aria Digital Twin: A New Benchmark Dataset for Egocentric 3D Machine Perception](https://mlsanity.com/paper/2306.06362/) (2023)
- [Rethinking Visual Geo-localization for Large-Scale Applications](https://mlsanity.com/paper/2204.02287/) (2022)

## Built on by (most cited first, 30 of 84)

- [Ego-Exo4D: Understanding Skilled Human Activity from First- and Third-Person Perspectives](https://mlsanity.com/paper/2311.18259/) (2023)
- [EgoExoLearn: A Dataset for Bridging Asynchronous Ego- and Exo-centric View of Procedural Activities in Real World](https://mlsanity.com/paper/2403.16182/) (2024)
- [EgoLife: Towards Egocentric Life Assistant](https://mlsanity.com/paper/2503.03803/) (2025)
- [Action Scene Graphs for Long-Form Understanding of Egocentric Videos](https://mlsanity.com/paper/2312.03391/) (2023)
- [PREGO: online mistake detection in PRocedural EGOcentric videos](https://mlsanity.com/paper/2404.01933/) (2024)
- [Learning Object State Changes in Videos: An Open-World Perspective](https://mlsanity.com/paper/2312.11782/) (2023)
- [EgoExo-Fitness: Towards Egocentric and Exocentric Full-Body Action Understanding](https://mlsanity.com/paper/2406.08877/) (2024)
- [EgoLM: Multi-Modal Language Model of Egocentric Motions](https://mlsanity.com/paper/2409.18127/) (2024)
- [Differentiable Task Graph Learning: Procedural Activity Representation and Online Mistake Detection from Egocentric Videos](https://mlsanity.com/paper/2406.01486/) (2024)
- [MM-Ego: Towards Building Egocentric Multimodal LLMs for Video QA](https://mlsanity.com/paper/2410.07177/) (2024)
- [EgoBlind: Towards Egocentric Visual Assistance for the Blind](https://mlsanity.com/paper/2503.08221/) (2025)
- [Modeling Fine-Grained Hand-Object Dynamics for Egocentric Video Representation Learning](https://mlsanity.com/paper/2503.00986/) (2025)
- [Challenges and Trends in Egocentric Vision: A Survey](https://mlsanity.com/paper/2503.15275/) (2025)
- [Vinci: A Real-time Embodied Smart Assistant based on Egocentric Vision-Language Model](https://mlsanity.com/paper/2412.21080/) (2024)
- [EAGLE: Egocentric AGgregated Language-video Engine](https://mlsanity.com/paper/2409.17523/) (2024)
- [X-LeBench: A Benchmark for Extremely Long Egocentric Video Understanding](https://mlsanity.com/paper/2501.06835/) (2025)
- [AFF-ttention! Affordances and Attention models for Short-Term Object Interaction Anticipation](https://mlsanity.com/paper/2406.01194/) (2024)
- [Reconstructing 4D Spatial Intelligence: A Survey](https://mlsanity.com/paper/2507.21045/) (2025)
- [A Survey on Deep Learning Techniques for Action Anticipation](https://mlsanity.com/paper/2309.17257/) (2023)
- [Do Egocentric Video-Language Models Truly Understand Hand-Object Interactions?](https://mlsanity.com/paper/2405.17719/) (2024)
- [Online Episodic Memory Visual Query Localization with Egocentric Streaming Object Memory](https://mlsanity.com/paper/2411.16934/) (2024)
- [Fine-grained Spatiotemporal Grounding on Egocentric Videos](https://mlsanity.com/paper/2508.00518/) (2025)
- [Developing Vision-Language-Action Model from Egocentric Videos](https://mlsanity.com/paper/2509.21986/) (2025)
- [A Backpack Full of Skills: Egocentric Video Understanding with Diverse Task Perspectives](https://mlsanity.com/paper/2403.03037/) (2024)
- [MEgoHand: Multimodal Egocentric Hand-Object Interaction Motion Generation](https://mlsanity.com/paper/2505.16602/) (2025)
- [An Egocentric Vision-Language Model based Portable Real-time Smart Assistant](https://mlsanity.com/paper/2503.04250/) (2025)
- [Bridging Perspectives: A Survey on Cross-view Collaborative Intelligence with Egocentric-Exocentric Vision](https://mlsanity.com/paper/2506.06253/) (2025)
- [MICA: Multi-Agent Industrial Coordination Assistant](https://mlsanity.com/paper/2509.15237/) (2025)
- [EgoControl: Controllable Egocentric Video Generation via 3D Full-Body Poses](https://mlsanity.com/paper/2511.18173/) (2025)
- [ToG-Bench: Task-Oriented Spatio-Temporal Grounding in Egocentric Videos](https://mlsanity.com/paper/2512.03666/) (2025)

## Abstract (arXiv)

What will the future be? We wonder! In this survey, we explore the gap between current research in egocentric vision and the ever-anticipated future, where wearable computing, with outward facing cameras and digital overlays, is expected to be integrated in our every day lives. To understand this gap, the article starts by envisaging the future through character-based stories, showcasing through examples the limitations of current technology. We then provide a mapping between this future and previously defined research tasks. For each task, we survey its seminal works, current state-of-the-art methodologies and available datasets, then reflect on shortcomings that limit its applicability to future research. Note that this survey focuses on software models for egocentric vision, independent of any specific hardware. The paper concludes with recommendations for areas of immediate explorations so as to unlock our path to the future always-on, personalised and life-enhancing egocentric vision.