# From Seeing to Acting: Smart Glasses as First-Person Intelligence Platforms

arXiv: https://arxiv.org/abs/2608.24877 · HTML: https://mlsanity.com/paper/2608.24877/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 81 of its 215 references that are in the map.
- First posted on arXiv 25 Aug 2026.

## Builds on (most cited first, 30 of 81)

- [Gemini 2.5: Pushing the Frontier with Advanced Reasoning, Multimodality, Long Context, and Next Generation Agentic Capabilities](https://mlsanity.com/paper/2507.06261/) (2025)
- [OpenVLA: An Open-Source Vision-Language-Action Model](https://mlsanity.com/paper/2406.09246/) (2024)
- [π0: A Vision-Language-Action Flow Model for General Robot Control](https://mlsanity.com/paper/2410.24164/) (2024)
- [Do As I Can, Not As I Say: Grounding Language in Robotic Affordances](https://mlsanity.com/paper/2204.01691/) (2022)
- [RT-1: Robotics Transformer for Real-World Control at Scale](https://mlsanity.com/paper/2212.06817/) (2022)
- [Matterport3D: Learning from RGB-D Data in Indoor Environments](https://mlsanity.com/paper/1709.06158/) (2017)
- [DROID: A Large-Scale In-The-Wild Robot Manipulation Dataset](https://mlsanity.com/paper/2403.12945/) (2024)
- [R3M: A Universal Visual Representation for Robot Manipulation](https://mlsanity.com/paper/2203.12601/) (2022)
- [GPT-4V(ision) is a Generalist Web Agent, if Grounded](https://mlsanity.com/paper/2401.01614/) (2024)
- [Mobile-Agent: Autonomous Multi-Modal Mobile Device Agent with Visual Perception](https://mlsanity.com/paper/2401.16158/) (2024)
- [Project Aria: A New Tool for Egocentric Multi-Modal AI Research](https://mlsanity.com/paper/2308.13561/) (2023)
- [Qwen3.5-Omni Technical Report](https://mlsanity.com/paper/2604.15804/) (2026)
- [EgoVLA: Learning Vision-Language-Action Models from Egocentric Human Videos](https://mlsanity.com/paper/2507.12440/) (2025)
- [EgoScale: Scaling Dexterous Manipulation with Diverse Egocentric Human Data](https://mlsanity.com/paper/2602.16710/) (2026)
- [Scalable Vision-Language-Action Model Pretraining for Robotic Manipulation with Real-Life Human Activity Videos](https://mlsanity.com/paper/2510.21571/) (2025)
- [EgoVerse: An Egocentric Human Dataset for Robot Learning from Around the World](https://mlsanity.com/paper/2604.07607/) (2026)
- [Qwen-RobotManip Technical Report: Alignment Unlocks Scale for Robotic Manipulation Foundation Models](https://mlsanity.com/paper/2606.17846/) (2026)
- [EgoZero: Robot Learning from Smart Glasses](https://mlsanity.com/paper/2505.20290/) (2025)
- [AI for Service: Proactive Assistance with AI Glasses](https://mlsanity.com/paper/2510.14359/) (2025)
- [EgoLive: A Large-Scale Egocentric Dataset from Real-World Human Tasks](https://mlsanity.com/paper/2604.23570/) (2026)
- [Dexterity from Smart Lenses: Multi-Fingered Robot Manipulation with In-the-Wild Human Demonstrations](https://mlsanity.com/paper/2511.16661/) (2025)
- [Latent Spatial Memory for Video World Models](https://mlsanity.com/paper/2606.09828/) (2026)
- [HumanEgo: Zero-Shot Robot Learning from Minutes of Human Egocentric Videos](https://mlsanity.com/paper/2605.24934/) (2026)
- [EgoMI: Learning Active Vision and Whole-Body Manipulation from Egocentric Human Demonstrations](https://mlsanity.com/paper/2511.00153/) (2025)
- [UniDex: A Robot Foundation Suite for Universal Dexterous Hand Control from Egocentric Human Videos](https://mlsanity.com/paper/2603.22264/) (2026)
- [EgoMemReason: A Memory-Driven Reasoning Benchmark for Long-Horizon Egocentric Video Understanding](https://mlsanity.com/paper/2605.09874/) (2026)
- [EgoForge: Goal-Directed Egocentric World Simulator](https://mlsanity.com/paper/2603.20169/) (2026)
- [TeleEgo: Benchmarking Egocentric AI Assistants in the Wild](https://mlsanity.com/paper/2510.23981/) (2025)
- [HumanNet: Scaling Human-centric Video Learning to One Million Hours](https://mlsanity.com/paper/2605.06747/) (2026)
- [Ego2Robot: Scalable Robot Data Synthesis from Egocentric Human Data](https://mlsanity.com/paper/2608.02580/) (2026)

## Abstract (arXiv)

Smart glasses are evolving from capture and display accessories into first-person intelligence platforms that connect human perception, persistent context, and digital or physical action. Their on-body viewpoint aligns with the wearer's vision, audition, motion, and hand-object interaction, but must operate under tight energy, thermal, privacy, and feedback constraints. Despite rapid progress in augmented reality, egocentric vision, multimodal models, human-computer interaction, and embodied intelligence, the literature remains fragmented across devices, tasks, and benchmarks. textitThe key challenge is not whether a model can recognize, answer, remember, or act in isolation, but whether a complete system can sustain a reliable, temporally valid, correctable, and governable perception-state-interaction-action loop. This survey is textitthe first to systematically study smart glasses through such a unified framework. We formalize first-person data flow and constrained task utility, characterize devices along eight verifiable hardware capability axes, organize the literature around seven interdependent foundational capabilities, and introduce an L0-L5 framework spanning capture, reactive perception, contextual assistance, persistent state, governed action, and embodied coupling. Across nine application scenes, we connect tasks with datasets, systems, products, stakeholders, failure consequences, and evidence gaps. We further present a nine-dimensional deployment framework, a claim-conditioned evaluation protocol, and an evidence ladder from controlled measurement to longitudinal field validation and audit. Together, these elements make smart glasses more comparable, deployable, and reproducibly evaluated, while outlining a roadmap toward trustworthy first-person intelligence.