mlsanityData to 14 Oct 2026

HOT3D: Hand and Object Tracking in 3D from Egocentric Multi-View Videos

arXiv 2411.19167 ↗First posted 28 Nov 2024cs.CV cs.AI cs.ROcited by 129

Not on the map. Why?

Abstract (arXiv)

We introduce HOT3D, a publicly available dataset for egocentric hand and object tracking in 3D. The dataset offers over 833 minutes (3.7M+ images) of recordings that feature 19 subjects interacting with 33 diverse rigid objects. In addition to simple pick-up, observe, and put-down actions, the subjects perform actions typical for a kitchen, office, and living room environment. The recordings include multiple synchronized data streams containing egocentric multi-view RGB/monochrome images, eye gaze signal, scene point clouds, and 3D poses of cameras, hands, and objects. The dataset is recorded with two headsets from Meta: Project Aria, which is a research prototype of AI glasses, and Quest 3, a virtual-reality headset that has shipped millions of units. Ground-truth poses were obtained by a motion-capture system using small optical markers attached to hands and objects. Hand annotations are provided in the UmeTrack and MANO formats, and objects are represented by 3D meshes with PBR materials obtained by an in-house scanner. In our experiments, we demonstrate the effectiveness of multi-view egocentric data for three popular tasks: 3D hand tracking, model-based 6DoF object pose estimation, and 3D lifting of unknown in-hand objects. The evaluated multi-view methods, whose benchmarking is uniquely enabled by HOT3D, significantly outperform their single-view counterparts.

Key facts. Cited by 129 papers in the map (data through 14 Oct 2026). Builds on 46 of its 81 references that are in the map. First posted on arXiv 28 Nov 2024.

Builds on

46 of 81 references.

References · 46

  1. 2014ImageNet Large Scale Visual Recognition Challengecited by 8,498
  2. 2023DINOv2: Learning Robust Visual Features without Supervisioncited by 7,974
  3. 2017Mask R-CNNcited by 7,277
  4. 2024Depth Anything V2cited by 1,422
  5. 2017PoseCNN: A Convolutional Neural Network for 6D Object Pose Estimation in Cluttered Scenescited by 775
  6. 2020Rescaling Egocentric Visioncited by 540
  7. 2019Learning joint reconstruction of hands and manipulated objectscited by 378
  8. 2022HOI4D: A 4D Egocentric Dataset for Category-Level Human-Object Interactioncited by 327
  9. 2021DexYCB: A Benchmark for Capturing Hand Grasping of Objectscited by 325
  10. 2022ARCTIC: A Dataset for Dexterous Bimanual Hand-Object Manipulationcited by 312
  11. 2020Understanding Human Hands in Contact at Internet Scalecited by 281
  12. 2019FreiHAND: A Dataset for Markerless Capture of Hand Pose and Shape from Single RGB Imagescited by 271
34 more
  1. 2018BOP: Benchmark for 6D Object Pose Estimationcited by 254
  2. 2022BEHAVE: Dataset and Method for Tracking Human Object Interactionscited by 249
  3. 2020InterHand2.6M: A Dataset and Baseline for 3D Interacting Hand Pose Estimation from a Single RGB Imagecited by 245
  4. 2017First-Person Hand Action Benchmark with RGB-D Videos and 3D Hand Pose Annotationscited by 230
  5. 2017T-LESS: An RGB-D Dataset for 6D Pose Estimation of Texture-less Objectscited by 210
  6. 2023Project Aria: A New Tool for Egocentric Multi-Modal AI Researchcited by 208
  7. 2020BOP Challenge 2020 on 6D Object Localizationcited by 193
  8. 2020ContactPose: A Dataset of Grasps with Object Contact and Hand Posecited by 183
  9. 2022OnePose: One-Shot Object Pose Estimation without CAD Modelscited by 157
  10. 2021Keypoint Transformer: Solving Joint Identification in Challenging Hands and Object Interactions for Accurate 3D Pose Estimationcited by 123
  11. 2015Capturing Hands in Action using Discriminative Salient Points and Physics Simulationcited by 115
  12. 2023FoundPose: Unseen Object Pose Estimation with Foundation Featurescited by 109
  13. 2017Real-time Hand Tracking under Occlusion from an Egocentric RGB-D Sensorcited by 84
  14. 2022Fine-Grained Egocentric Hand-Object Segmentation: Dataset, Model, and Applicationscited by 78
  15. 2023BOP Challenge 2022 on Detection, Segmentation and Pose Estimation of Specific Rigid Objectscited by 70
  16. 2019Photorealistic Image Synthesis for Object Instance Detectioncited by 66
  17. 2023AssemblyHands: Towards Egocentric Activity Understanding via 3D Hand Pose Estimationcited by 64
  18. 2024BOP Challenge 2023 on Detection, Segmentation and Pose Estimation of Seen and Unseen Rigid Objectscited by 60
  19. 2022Category-Level 6D Object Pose Estimation in the Wild: A Semi-Supervised Learning Approach and A New Datasetcited by 59
  20. 2022Zero-Shot Category-Level Object Pose Estimationcited by 57
  21. 2016Real-time Joint Tracking of a Hand Manipulating an Object from RGB-D Inputcited by 56
  22. 2022HouseCat6D -- A Large-Scale Multi-Modal Category Level 6D Object Perception Dataset with Household Objects in Realistic Scenarioscited by 53
  23. 2022UmeTrack: Unified multi-view end-to-end hand tracking for VRcited by 50
  24. 2023HANDAL: A Dataset of Real-World Manipulable Object Categories with Pose Annotations, Affordances, and Reconstructionscited by 48
  25. 2022PhoCaL: A Multi-Modal Dataset for Category-Level Object Pose Estimation with Photometrically Challenging Objectscited by 45
  26. 20163D Hand Pose Tracking and Estimation Using Stereo Matchingcited by 35
  27. 2020HoloLens 2 Research Mode as a Tool for Computer Vision Researchcited by 34
  28. 2024HO-Cap: A Capture System and Dataset for 3D Reconstruction and Pose Tracking of Hand-Object Interactioncited by 29
  29. 2024Spatial Cognition from Egocentric Video: Out of Sight, Not Out of Mindcited by 28
  30. 2022In-Hand 3D Object Scanning from an RGB Sequencecited by 27
  31. 2024Dense Hand-Object(HO) GraspNet with Full Grasping Taxonomy and Dynamicscited by 15
  32. 2023ENIGMA-51: Towards a Fine-Grained Understanding of Human-Object Interactions in Industrial Scenarioscited by 11
  33. 2017The 2017 Hands in the Million Challenge on 3D Hand Pose Estimationcited by 6
  34. 2018HANDS18: Methods, Techniques and Applications for Hand Observationcited by 4

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 2; 2025: 41; 2026: 86.

129 papers.

Sort:

By area:

  1. 2025EgoDex: Learning Dexterous Manipulation from Large-Scale Egocentric Videocited by 211
  2. 2025HD-EPIC: A Highly-Detailed Egocentric Video Datasetcited by 104
  3. 2025Being-H0: Vision-Language-Action Pretraining from Large-Scale Human Videoscited by 102
  4. 2025EgoLife: Towards Egocentric Life Assistantcited by 91
  5. 2025Vision-Language-Action Models for Robotics: A Review Towards Real-World Applicationscited by 84
  6. 2024Dyn-HaMR: Recovering 4D Interacting Hand Motion from a Dynamic Cameracited by 51
  7. 2026World Action Models: The Next Frontier in Embodied AIcited by 50
  8. 20254DGT: Learning a 4D Gaussian Transformer Using Real-World Monocular Videoscited by 49
  9. 2025H-RDT: Human Manipulation Enhanced Bimanual Robotic Manipulationcited by 45
  10. 2025DexWild: Dexterous Human Interactions for In-the-Wild Robot Policiescited by 38
  11. 2025SPIDER: Scalable Physics-Informed Dexterous Retargetingcited by 32
  12. 2025World Models for Learning Dexterous Hand-Object Interactions from Human Videoscited by 31
38 more
  1. 2025In-N-On: Scaling Egocentric Manipulation with in-the-wild and on-task Datacited by 26
  2. 2026RoboLab: A High-Fidelity Simulation Benchmark for Analysis of Task Generalist Policiescited by 24
  3. 2025BOP Challenge 2024 on Model-Based and Model-Free 6D Object Pose Estimationcited by 19
  4. 2025Dexterity from Smart Lenses: Multi-Fingered Robot Manipulation with In-the-Wild Human Demonstrationscited by 19
  5. 2026LAP: Language-Action Pre-Training Enables Zero-shot Cross-Embodiment Transfercited by 18
  6. 2026Hand2World: Autoregressive Egocentric Interaction Generation via Free-Space Hand Gesturescited by 16
  7. 2025OPENTOUCH: Bringing Full-Hand Touch to Real-World Interactioncited by 15
  8. 2026Joint-Aligned Latent Action: Towards Scalable VLA Pretraining in the Wildcited by 15
  9. 2025CoDA: Coordinated Diffusion Noise Optimization for Whole-Body Manipulation of Articulated Objectscited by 12
  10. 2025OSMO: Open-Source Tactile Glove for Human-to-Robot Skill Transfercited by 10
  11. 2025MEgoHand: Multimodal Egocentric Hand-Object Interaction Motion Generationcited by 9
  12. 2025The Monado SLAM Dataset for Egocentric Visual-Inertial Trackingcited by 9
  13. 2025Novel Diffusion Models for Multimodal 3D Hand Trajectory Predictioncited by 8
  14. 2025DyTact: Capturing Dynamic Contacts in Hand-Object Manipulationcited by 8
  15. 2026WHOLE: World-Grounded Hand-Object Lifted from Egocentric Videoscited by 8
  16. 2026FunRec: Reconstructing Functional 3D Scenes from Egocentric Interaction Videoscited by 8
  17. 2026Open-AoE: An Open Egocentric Manipulation Dataset and Toolchain for Embodied Learningcited by 8
  18. 2026ForeHOI: Feed-forward 3D Object Reconstruction from Daily Hand-Object Interaction Videoscited by 7
  19. 2026FlowHOI: Flow-based Semantics-Grounded Generation of Hand-Object Interactions for Dexterous Robot Manipulationcited by 7
  20. 2025Aria Gen 2 Pilot Datasetcited by 6
  21. 2026ActiveMimic: Egocentric Video Pretraining with Active Perceptioncited by 6
  22. 2026Learning Dexterous Manipulation Using Contact Wrench Guidance From Human Demonstrationcited by 6
  23. 2025EventEgoHands: Event-based Egocentric 3D Hand Mesh Reconstructioncited by 5
  24. 2025ECHO: Ego-Centric modeling of Human-Object interactionscited by 5
  25. 2025Uni-Hand: Universal Hand Motion Forecasting in Egocentric Viewscited by 5
  26. 2025GR-Dexter Technical Reportcited by 5
  27. 2026ObjectForesight: Predicting Future 3D Object Trajectories from Human Videoscited by 5
  28. 2026Robot Learning from Human Videos: A Surveycited by 5
  29. 2026MolmoMotion: Forecasting Point Trajectories in 3D with Language Instructioncited by 5
  30. 2026The Surprising Effectiveness of Video Diffusion Models for Hand Motion Reconstructioncited by 5
  31. 2026Human-Centric Transferable Tactile Pre-Training for Dexterous Robotic Manipulationcited by 5
  32. 2026HandEdit: A Unified Benchmark for Egocentric Human-to-Robot Dexterous Hand Image Editingcited by 5
  33. 2026SHOW3D: Capturing Scenes of 3D Hands and Objects in the Wildcited by 4
  34. 2026HRDexDB: A 4D Dexterous Grasping Dataset Across Human and Multiple Robot Embodimentscited by 4
  35. 2026EgoTactile: Learning Grasp Pressure for Everyday Objects from Egocentric Videocited by 4
  36. 2024GFreeDet: Exploiting Gaussian Splatting and Foundation Models for Model-free Unseen Object Detection in the BOP Challenge 2024cited by 3
  37. 2025Towards Dynamic 3D Reconstruction of Hand-Instrument Interaction in Ophthalmic Surgerycited by 3
  38. 2025PALM: A Dataset and Baseline for Learning Multi-subject Hand Priorcited by 3

50 most cited of 129 All 129 (JSON)