# Dexterity from Smart Lenses: Multi-Fingered Robot Manipulation with In-the-Wild Human Demonstrations

arXiv: https://arxiv.org/abs/2511.16661 · HTML: https://mlsanity.com/paper/2511.16661/

## Key facts

- Cited by 19 papers in the map (data through 14 Oct 2026).
- Builds on 54 of its 64 references that are in the map.
- First posted on arXiv 20 Nov 2025.

## Builds on (most cited first, 30 of 54)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Masked Autoencoders Are Scalable Vision Learners](https://mlsanity.com/paper/2111.06377/) (2021)
- [RT-1: Robotics Transformer for Real-World Control at Scale](https://mlsanity.com/paper/2212.06817/) (2022)
- [Ego4D: Around the World in 3,000 Hours of Egocentric Video](https://mlsanity.com/paper/2110.07058/) (2021)
- [Grounded SAM: Assembling Open-World Models for Diverse Visual Tasks](https://mlsanity.com/paper/2401.14159/) (2024)
- [Solving Rubik's Cube with a Robot Hand](https://mlsanity.com/paper/1910.07113/) (2019)
- [R3M: A Universal Visual Representation for Robot Manipulation](https://mlsanity.com/paper/2203.12601/) (2022)
- [Perceiver-Actor: A Multi-Task Transformer for Robotic Manipulation](https://mlsanity.com/paper/2209.05451/) (2022)
- [CoTracker: It is Better to Track Together](https://mlsanity.com/paper/2307.07635/) (2023)
- [Ego-Exo4D: Understanding Skilled Human Activity from First- and Third-Person Perspectives](https://mlsanity.com/paper/2311.18259/) (2023)
- [FoundationPose: Unified 6D Pose Estimation and Tracking of Novel Objects](https://mlsanity.com/paper/2312.08344/) (2023)
- [Unleashing Large-Scale Video Generative Pre-training for Visual Robot Manipulation](https://mlsanity.com/paper/2312.13139/) (2023)
- [Reconstructing Hands in 3D with Transformers](https://mlsanity.com/paper/2312.05251/) (2023)
- [Vector Neurons: A General Framework for SO(3)-Equivariant Networks](https://mlsanity.com/paper/2104.12229/) (2021)
- [TAP-Vid: A Benchmark for Tracking Any Point in a Video](https://mlsanity.com/paper/2211.03726/) (2022)
- [Masked Visual Pre-training for Motor Control](https://mlsanity.com/paper/2203.06173/) (2022)
- [Affordances from Human Videos as a Versatile Representation for Robotics](https://mlsanity.com/paper/2304.08488/) (2023)
- [MimicPlay: Long-Horizon Imitation Learning by Watching Human Play](https://mlsanity.com/paper/2302.12422/) (2023)
- [Project Aria: A New Tool for Egocentric Multi-Modal AI Research](https://mlsanity.com/paper/2308.13561/) (2023)
- [Putting the Object Back into Video Object Segmentation](https://mlsanity.com/paper/2310.12982/) (2023)
- [EgoMimic: Scaling Imitation Learning via Egocentric Video](https://mlsanity.com/paper/2410.24221/) (2024)
- [FoundationStereo: Zero-Shot Stereo Matching](https://mlsanity.com/paper/2501.09898/) (2025)
- [Language-Driven Representation Learning for Robotics](https://mlsanity.com/paper/2302.12766/) (2023)
- [DexCap: Scalable and Portable Mocap Data Collection System for Dexterous Manipulation](https://mlsanity.com/paper/2403.07788/) (2024)
- [Act3D: 3D Feature Field Transformers for Multi-Task Robotic Manipulation](https://mlsanity.com/paper/2306.17817/) (2023)
- [HOT3D: Hand and Object Tracking in 3D from Egocentric Multi-View Videos](https://mlsanity.com/paper/2411.19167/) (2024)
- [CoTracker3: Simpler and Better Point Tracking by Pseudo-Labelling Real Videos](https://mlsanity.com/paper/2410.11831/) (2024)
- [Track2Act: Predicting Point Tracks from Internet Videos enables Generalizable Robot Manipulation](https://mlsanity.com/paper/2405.01527/) (2024)
- [VideoDex: Learning Dexterity from Internet Videos](https://mlsanity.com/paper/2212.04498/) (2022)
- [GNFactor: Multi-Task Real Robot Learning with Generalizable Neural Feature Fields](https://mlsanity.com/paper/2308.16891/) (2023)

## Built on by (most cited first, 19 of 19)

- [EgoVerse: An Egocentric Human Dataset for Robot Learning from Around the World](https://mlsanity.com/paper/2604.07607/) (2026)
- [HumanEgo: Zero-Shot Robot Learning from Minutes of Human Egocentric Videos](https://mlsanity.com/paper/2605.24934/) (2026)
- [Do as I Do: Dexterous Manipulation Data from Everyday Human Videos](https://mlsanity.com/paper/2606.19333/) (2026)
- [Dexterous Manipulation Policies from RGB Human Videos via 3D Hand-Object Trajectory Reconstruction](https://mlsanity.com/paper/2602.09013/) (2026)
- [OSMO: Open-Source Tactile Glove for Human-to-Robot Skill Transfer](https://mlsanity.com/paper/2512.08920/) (2025)
- [World-Action Models for Robot Learning and Control: A Survey](https://mlsanity.com/paper/2609.16074/) (2026)
- [Robot Learning from Human Videos: A Survey](https://mlsanity.com/paper/2604.27621/) (2026)
- [LUCID: Learning Embodiment-Agnostic Intent Models from Unstructured Human Videos for Scalable Dexterous Robot Skill Acquisition](https://mlsanity.com/paper/2606.11628/) (2026)
- [Human Universal Grasping](https://mlsanity.com/paper/2606.17054/) (2026)
- [StableHand: Quality-Aware Flow Matching for World-Space Dual-Hand Motion Estimation from Egocentric Video](https://mlsanity.com/paper/2605.18553/) (2026)
- [Ruka-v2: Tendon Driven Open-Source Dexterous Hand with Wrist and Abduction for Robot Learning](https://mlsanity.com/paper/2603.26660/) (2026)
- [CRAFT: A Tendon-Driven Hand with Hybrid Hard-Soft Compliance](https://mlsanity.com/paper/2603.12120/) (2026)
- [HandelBot: Real-World Piano Playing via Fast Adaptation of Dexterous Robot Policies](https://mlsanity.com/paper/2603.12243/) (2026)
- [KPGrasp: Scalable Keypoint Flow Matching for Dexterous Grasp Generation](https://mlsanity.com/paper/2606.09314/) (2026)
- [From Seeing to Acting: Smart Glasses as First-Person Intelligence Platforms](https://mlsanity.com/paper/2608.24877/) (2026)
- [Track, Articulate, Act: Generating Articulation from Casual Human Videos](https://mlsanity.com/paper/2609.19119/) (2026)
- [Skel-WAM: A Hand-Skeleton-Conditioned World Action Model for Human-to-Robot Manipulation Transfer](https://mlsanity.com/paper/2609.21514/) (2026)
- [EigenDEXplore: Structured Exploration for Dexterous Manipulation with Human Priors](https://mlsanity.com/paper/2610.07681/) (2026)
- [Steertuning Policies from a Few Human Videos](https://mlsanity.com/paper/2610.18711/) (2026)

## Abstract (arXiv)

Learning multi-fingered robot policies from humans performing daily tasks in natural environments has long been a grand goal in the robotics community. Achieving this would mark significant progress toward generalizable robot manipulation in human environments, as it would reduce the reliance on labor-intensive robot data collection. Despite substantial efforts, progress toward this goal has been bottle-necked by the embodiment gap between humans and robots, as well as by difficulties in extracting relevant contextual and motion cues that enable learning of autonomous policies from in-the-wild human videos. We claim that with simple yet sufficiently powerful hardware for obtaining human data and our proposed framework AINA, we are now one significant step closer to achieving this dream. AINA enables learning multi-fingered policies from data collected by anyone, anywhere, and in any environment using Aria Gen 2 glasses. These glasses are lightweight and portable, feature a high-resolution RGB camera, provide accurate on-board 3D head and hand poses, and offer a wide stereo view that can be leveraged for depth estimation of the scene. This setup enables the learning of 3D point-based policies for multi-fingered hands that are robust to background changes and can be deployed directly without requiring any robot data (including online corrections, reinforcement learning, or simulation). We compare our framework against prior human-to-robot policy learning approaches, ablate our design choices, and demonstrate results across nine everyday manipulation tasks. Robot rollouts are best viewed on our website: https://aina-robot.github.io.