# EgoAERO: Learning Dexterous Manipulation from a Single Egocentric Video without Object Assets

arXiv: https://arxiv.org/abs/2606.08057 · HTML: https://mlsanity.com/paper/2606.08057/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 22 of its 25 references that are in the map.
- First posted on arXiv 6 Jun 2026.

## Builds on (most cited first, 22 of 22)

- [Segment Anything](https://mlsanity.com/paper/2304.02643/) (2023)
- [Ego4D: Around the World in 3,000 Hours of Egocentric Video](https://mlsanity.com/paper/2110.07058/) (2021)
- [Isaac Gym: High Performance GPU-Based Physics Simulation For Robot Learning](https://mlsanity.com/paper/2108.10470/) (2021)
- [SAM 3: Segment Anything with Concepts](https://mlsanity.com/paper/2511.16719/) (2025)
- [ORB-SLAM3: An Accurate Open-Source Library for Visual, Visual-Inertial and Multi-Map SLAM](https://mlsanity.com/paper/2007.11898/) (2020)
- [Scaling Egocentric Vision: The EPIC-KITCHENS Dataset](https://mlsanity.com/paper/1804.02748/) (2018)
- [FoundationPose: Unified 6D Pose Estimation and Tracking of Novel Objects](https://mlsanity.com/paper/2312.08344/) (2023)
- [Universal Manipulation Interface: In-The-Wild Robot Teaching Without In-The-Wild Robots](https://mlsanity.com/paper/2402.10329/) (2024)
- [HOI4D: A 4D Egocentric Dataset for Category-Level Human-Object Interaction](https://mlsanity.com/paper/2203.01577/) (2022)
- [DexYCB: A Benchmark for Capturing Hand Grasping of Objects](https://mlsanity.com/paper/2104.04631/) (2021)
- [ARCTIC: A Dataset for Dexterous Bimanual Hand-Object Manipulation](https://mlsanity.com/paper/2204.13662/) (2022)
- [H2O: Two Hands Manipulating Objects for First Person Interaction Recognition](https://mlsanity.com/paper/2104.11181/) (2021)
- [SAM 3D: 3Dfy Anything in Images](https://mlsanity.com/paper/2511.16624/) (2025)
- [EgoDex: Learning Dexterous Manipulation from Large-Scale Egocentric Video](https://mlsanity.com/paper/2505.11709/) (2025)
- [DexMV: Imitation Learning for Dexterous Manipulation from Human Videos](https://mlsanity.com/paper/2108.05877/) (2021)
- [BundleSDF: Neural 6-DoF Tracking and 3D Reconstruction of Unknown Objects](https://mlsanity.com/paper/2303.14158/) (2023)
- [OakInk: A Large-scale Knowledge Repository for Understanding Hand-Object Interaction](https://mlsanity.com/paper/2203.15709/) (2022)
- [HOT3D: Hand and Object Tracking in 3D from Egocentric Multi-View Videos](https://mlsanity.com/paper/2411.19167/) (2024)
- [BundleTrack: 6D Pose Tracking for Novel Objects without Instance or Category-Level 3D Models](https://mlsanity.com/paper/2108.00516/) (2021)
- [HaWoR: World-Space Hand Motion Reconstruction from Egocentric Videos](https://mlsanity.com/paper/2501.02973/) (2025)
- [ManipTrans: Efficient Dexterous Bimanual Manipulation Transfer via Residual Learning](https://mlsanity.com/paper/2503.21860/) (2025)
- [FastUMI: A Scalable and Hardware-Independent Universal Manipulation Interface with Dataset](https://mlsanity.com/paper/2409.19499/) (2024)

## Built on by (most cited first, 2 of 2)

- [Open-AoE: An Open Egocentric Manipulation Dataset and Toolchain for Embodied Learning](https://mlsanity.com/paper/2607.14183/) (2026)
- [MEMORA: Embodied Action Memory from Egocentric Videos for Reasoning and Planning](https://mlsanity.com/paper/2607.14252/) (2026)

## Abstract (arXiv)

Egocentric RGB-D videos offer a natural source of human dexterous manipulation demonstrations, but existing data is difficult to use for robot learning because object pose, geometry, and contact information are often missing or require pre-scanned object assets. We present EgoAERO, the first framework that learns dexterous manipulation from a single egocentric RGB-D human demonstration without object assets. EgoAERO reconstructs contact-consistent hand-object trajectories through asset-free object tracking and reconstruction, ego motion compensation, and adaptive contact optimization, then converts them into robot policies using two-stage residual learning. We further introduce an online quality assessment mechanism and construct EgoDex-R, a large-scale egocentric dataset with 4.3M RGB-D frames for dexterous policy learning. Simulation and real-world experiments show that EgoAERO enables single-demonstration dexterous manipulation and achieves downstream performance close to CAD-based reconstructions on HOI4D.