mlsanityData to 7 Oct 2026

EgoDex: Learning Dexterous Manipulation from Large-Scale Egocentric Video

arXiv 2505.11709 ↗First posted 16 May 2025cs.CV cs.LG cs.ROcited by 191

Not on the map. Why?

Cited by 55 in Vision-language-action models, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

Imitation learning for manipulation has a well-known data scarcity problem. Unlike natural language and 2D computer vision, there is no Internet-scale corpus of data for dexterous manipulation. One appealing option is egocentric human video, a passively scalable data source. However, existing large-scale datasets such as Ego4D do not have native hand pose annotations and do not focus on object manipulation. To this end, we use Apple Vision Pro to collect EgoDex: the largest and most diverse dataset of dexterous human manipulation to date. EgoDex has 829 hours of egocentric video with paired 3D hand and finger tracking data collected at the time of recording, where multiple calibrated cameras and on-device SLAM can be used to precisely track the pose of every joint of each hand. The dataset covers a wide range of diverse manipulation behaviors with everyday household objects in 194 different tabletop tasks ranging from tying shoelaces to folding laundry. Furthermore, we train and systematically evaluate imitation learning policies for hand trajectory prediction on the dataset, introducing metrics and benchmarks for measuring progress in this increasingly important area. By releasing this large-scale dataset, we hope to push the frontier of robotics, computer vision, and foundation models. EgoDex is publicly available for download at https://github.com/apple/ml-egodex.

Key facts. Cited by 191 papers in the map (data through 7 Oct 2026). Builds on 37 of its 45 references that are in the map. First posted on arXiv 16 May 2025.

Builds on

37 of 45 references.

References · 37

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  2. 2014ImageNet Large Scale Visual Recognition Challengecited by 8,441
  3. 2019Dream to Control: Learning Behaviors by Latent Imaginationcited by 1,598
  4. 2021Ego4D: Around the World in 3,000 Hours of Egocentric Videocited by 1,530
  5. 2018On the Continuity of Rotation Representations in Neural Networkscited by 1,186
  6. 2023Open X-Embodiment: Robotic Learning Datasets and RT-X Modelscited by 849
  7. 2025Cosmos World Foundation Model Platform for Physical AIcited by 833
  8. 2018Scaling Egocentric Vision: The EPIC-KITCHENS Datasetcited by 789
  9. 2024Genie: Generative Interactive Environmentscited by 706
  10. 2022R3M: A Universal Visual Representation for Robot Manipulationcited by 678
  11. 2024Autoregressive Image Generation without Vector Quantizationcited by 644
  12. 2023BridgeData V2: A Dataset for Robot Learning at Scalecited by 634
25 more
  1. 2023Learning Interactive Real-World Simulatorscited by 440
  2. 2022VIP: Towards Universal Visual Reward and Representation via Value-Implicit Pre-Trainingcited by 384
  3. 2024Universal Manipulation Interface: In-The-Wild Robot Teaching Without In-The-Wild Robotscited by 373
  4. 2022HOI4D: A 4D Egocentric Dataset for Category-Level Human-Object Interactioncited by 321
  5. 2021DexYCB: A Benchmark for Capturing Hand Grasping of Objectscited by 320
  6. 2023Reconstructing Hands in 3D with Transformerscited by 298
  7. 2019RoboNet: Large-Scale Multi-Robot Learningcited by 268
  8. 2022Real-World Robot Learning with Masked Visual Pre-trainingcited by 241
  9. 2023Affordances from Human Videos as a Versatile Representation for Roboticscited by 221
  10. 2024EgoMimic: Scaling Imitation Learning via Egocentric Videocited by 160
  11. 2025Open-Sora 2.0: Training a Commercial-Level Video Generation Model in 200kcited by 157
  12. 2024DexCap: Scalable and Portable Mocap Data Collection System for Dexterous Manipulationcited by 149
  13. 2023FurnitureBench: Reproducible Real-World Benchmark for Long-Horizon Complex Manipulationcited by 132
  14. 2024HOT3D: Hand and Object Tracking in 3D from Egocentric Multi-View Videoscited by 122
  15. 2025Humanoid Policy ~ Human Policycited by 65
  16. 2025Motion Tracks: A Unified Representation for Human-Robot Transfer in Few-Shot Imitation Learningcited by 57
  17. 2024RoVi-Aug: Robot and Viewpoint Augmentation for Cross-Embodiment Robot Learningcited by 46
  18. 2025RoboEngine: Plug-and-Play Robot Data Augmentation with Semantic Robot Segmentation and Background Generationcited by 40
  19. 2024Hand-Object Interaction Pretraining from Videoscited by 38
  20. 2024ARCap: Collecting High-quality Human Demonstrations for Robot Learning with Augmented Reality Feedbackcited by 31
  21. 2025TASTE-Rob: Advancing Video Generation of Task-Oriented Hand-Object Interaction for Generalizable Robotic Manipulationcited by 24
  22. 2025Learning Real-World Action-Video Dynamics with Heterogeneous Masked Autoregressioncited by 21
  23. 2025MAPLE: Encoding Dexterous Robotic Manipulation Priors Learned From Egocentric Videoscited by 12
  24. 2024DexHub and DART: Towards Internet Scale Robot Data Collectioncited by 11
  25. 2024ARMADA: Augmented Reality for Robot Manipulation and Robot-Free Data Acquisitioncited by 4

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 40; 2026: 151.

191 papers.

Sort:

By area:

  1. 2026World Action Models are Zero-shot Policiescited by 411
  2. 2025Motus: A Unified Latent Action World Modelcited by 231
  3. 2026GigaWorld-Policy: An Efficient Action-Centered World--Action Modelcited by 129
  4. 2026DreamDojo: A Generalist Robot World Model from Large-Scale Human Videoscited by 117
  5. 2025GR-3 Technical Reportcited by 110
  6. 2026StarVLA: A Lego-like Codebase for Vision-Language-Action Model Developingcited by 103
  7. 2025Being-H0: Vision-Language-Action Pretraining from Large-Scale Human Videoscited by 94
  8. 2026Being-H0.7: A Latent World-Action Model from Egocentric Videoscited by 68
  9. 2025Large VLM-based Vision-Language-Action Models for Robotic Manipulation: A Surveycited by 66
  10. 2025A Vision-Language-Action-Critic Model for Robotic Real-World Reinforcement Learningcited by 66
  11. 2025Scalable Vision-Language-Action Model Pretraining for Robotic Manipulation with Real-Life Human Activity Videoscited by 63
  12. 2026Being-H0.5: Scaling Human-Centric Robot Learning for Cross-Embodiment Generalizationcited by 61
38 more
  1. 2025OmniWorld: A Multi-Domain and Multi-Modal Dataset for 4D World Modelingcited by 59
  2. 2025Vidar: Embodied Video Diffusion Model for Generalist Manipulationcited by 58
  3. 2026Robometer: Scaling General-Purpose Robotic Reward Models via Trajectory Comparisonscited by 51
  4. 2026EgoVerse: An Egocentric Human Dataset for Robot Learning from Around the Worldcited by 49
  5. 2026InternVLA-A1: Unifying Understanding, Generation and Action for Robotic Manipulationcited by 48
  6. 2026Qwen-RobotManip Technical Report: Alignment Unlocks Scale for Robotic Manipulation Foundation Modelscited by 47
  7. 2026World Action Models: The Next Frontier in Embodied AIcited by 46
  8. 2026LaWAM: Latent World Action Models for Efficient Dynamics-Aware Robot Policiescited by 46
  9. 2025GigaBrain-0: A World Model-Powered Vision-Language-Action Modelcited by 43
  10. 2026Learning Latent Action World Models In The Wildcited by 42
  11. 2026Qwen-VLA: Unifying Vision-Language-Action Modeling across Tasks, Environments, and Robot Embodimentscited by 42
  12. 2025H-RDT: Human Manipulation Enhanced Bimanual Robotic Manipulationcited by 41
  13. 2025Robo-Dopamine: General Process Reward Modeling for High-Precision Robotic Manipulationcited by 37
  14. 2026Infinite Worlds with Versatile Interactionscited by 37
  15. 2025A Survey on Efficient Vision-Language-Action Modelscited by 34
  16. 2025Dual-Stream Diffusion for World-Model Augmented Vision-Language-Action Modelcited by 33
  17. 2025World Models for Learning Dexterous Hand-Object Interactions from Human Videoscited by 26
  18. 2025SPIDER: Scalable Physics-Informed Dexterous Retargetingcited by 25
  19. 2026BagelVLA: Enhancing Long-Horizon Manipulation via Interleaved Vision-Language-Action Generationcited by 25
  20. 2025RynnVLA-001: Using Human Demonstrations to Improve Robot Manipulationcited by 24
  21. 2025Robix: A Unified Model for Robot Interaction, Reasoning and Planningcited by 23
  22. 2025In-N-On: Scaling Egocentric Manipulation with in-the-wild and on-task Datacited by 21
  23. 2026A Systematic Study of Data Modalities and Strategies for Co-training Large Behavior Models for Robot Manipulationcited by 20
  24. 2026HoloBrain-0 Technical Reportcited by 20
  25. 2026LAP: Language-Action Pre-Training Enables Zero-shot Cross-Embodiment Transfercited by 17
  26. 2026DIAL: Decoupling Intent and Action via Latent World Modeling for End-to-End VLAcited by 17
  27. 2026EgoWAM: World Action Models Beyond Pixels with In-the-Wild Egocentric Human Datacited by 17
  28. 2025Vidarc: Embodied Video Diffusion Model for Closed-loop Controlcited by 15
  29. 2025Dexterous World Modelscited by 15
  30. 2026HumanEgo: Zero-Shot Robot Learning from Minutes of Human Egocentric Videoscited by 14
  31. 2025OpenEgo: A Large-Scale Multimodal Egocentric Dataset for Dexterous Manipulationcited by 12
  32. 2026ThinkJEPA: Empowering Latent World Models with Large Vision-Language Reasoning Modelcited by 12
  33. 2025OPENTOUCH: Bringing Full-Hand Touch to Real-World Interactioncited by 11
  34. 2025SpatialTree: How Spatial Abilities Branch Out in MLLMscited by 11
  35. 2026ConLA: Contrastive Latent Action Learning from Human Videos for Robotic Manipulationcited by 11
  36. 2026Large Reward Models: Generalizable Online Robot Reward Generation with Vision-Language Modelscited by 11
  37. 2026EgoSim: Egocentric World Simulator for Embodied Interaction Generationcited by 11
  38. 2025Developing Vision-Language-Action Model from Egocentric Videoscited by 10

50 most cited of 191 All 191 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2026EgoScale: Scaling Dexterous Manipulation with Diverse Egocentric Human Data
  2. 2026Wh0: Generative World Models as Scalable Sources of Egocentric Human Hand Manipulation Data
  3. 2026UniDex: A Robot Foundation Suite for Universal Dexterous Hand Control from Egocentric Human Videos
  4. 2026EgoSteer: An Open-Source Full-Stack System Towards Steerable Dexterous Manipulation from Egocentric Videos
  5. 2026Ego2Robot: Scalable Robot Data Synthesis from Egocentric Human Data
  6. 2026IronMind: Scaling Humanoid Dexterous Manipulation via Camera-Space Ego-Centric Pretraining ↗
  7. 2021DexMV: Imitation Learning for Dexterous Manipulation from Human Videos
  8. 2026EaDex: A Cross-Embodiment Dexterous Manipulation Framework from Low-Cost Demonstrations