mlsanityData to 7 Oct 2026

Introducing HOT3D: An Egocentric Dataset for 3D Hand and Object Tracking

arXiv 2406.09598 ↗First posted 13 Jun 2024cs.CVcited by 46

Not on the map. Why?

Abstract (arXiv)

We introduce HOT3D, a publicly available dataset for egocentric hand and object tracking in 3D. The dataset offers over 833 minutes (more than 3.7M images) of multi-view RGB/monochrome image streams showing 19 subjects interacting with 33 diverse rigid objects, multi-modal signals such as eye gaze or scene point clouds, as well as comprehensive ground truth annotations including 3D poses of objects, hands, and cameras, and 3D models of hands and objects. In addition to simple pick-up/observe/put-down actions, HOT3D contains scenarios resembling typical actions in a kitchen, office, and living room environment. The dataset is recorded by two head-mounted devices from Meta: Project Aria, a research prototype of light-weight AR/AI glasses, and Quest 3, a production VR headset sold in millions of units. Ground-truth poses were obtained by a professional motion-capture system using small optical markers attached to hands and objects. Hand annotations are provided in the UmeTrack and MANO formats and objects are represented by 3D meshes with PBR materials obtained by an in-house scanner. We aim to accelerate research on egocentric hand-object interaction by making the HOT3D dataset publicly available and by co-organizing public challenges on the dataset at ECCV 2024. The dataset can be downloaded from the project website: https://facebookresearch.github.io/hot3d/.

Key facts. Cited by 46 papers in the map (data through 7 Oct 2026). Builds on 36 of its 62 references that are in the map. First posted on arXiv 13 Jun 2024.

Builds on

36 of 62 references.

References · 36

  1. 2014ImageNet Large Scale Visual Recognition Challengecited by 8,441
  2. 2017PoseCNN: A Convolutional Neural Network for 6D Object Pose Estimation in Cluttered Scenescited by 768
  3. 2020Rescaling Egocentric Visioncited by 529
  4. 2019Learning joint reconstruction of hands and manipulated objectscited by 377
  5. 2022HOI4D: A 4D Egocentric Dataset for Category-Level Human-Object Interactioncited by 321
  6. 2021DexYCB: A Benchmark for Capturing Hand Grasping of Objectscited by 320
  7. 2022ARCTIC: A Dataset for Dexterous Bimanual Hand-Object Manipulationcited by 306
  8. 2020Understanding Human Hands in Contact at Internet Scalecited by 281
  9. 2019FreiHAND: A Dataset for Markerless Capture of Hand Pose and Shape from Single RGB Imagescited by 271
  10. 2018BOP: Benchmark for 6D Object Pose Estimationcited by 254
  11. 2022BEHAVE: Dataset and Method for Tracking Human Object Interactionscited by 246
  12. 2020InterHand2.6M: A Dataset and Baseline for 3D Interacting Hand Pose Estimation from a Single RGB Imagecited by 244
24 more
  1. 2017First-Person Hand Action Benchmark with RGB-D Videos and 3D Hand Pose Annotationscited by 229
  2. 2017T-LESS: An RGB-D Dataset for 6D Pose Estimation of Texture-less Objectscited by 210
  3. 2023Project Aria: A New Tool for Egocentric Multi-Modal AI Researchcited by 201
  4. 2020BOP Challenge 2020 on 6D Object Localizationcited by 192
  5. 2020ContactPose: A Dataset of Grasps with Object Contact and Hand Posecited by 180
  6. 2022OnePose: One-Shot Object Pose Estimation without CAD Modelscited by 155
  7. 2021Keypoint Transformer: Solving Joint Identification in Challenging Hands and Object Interactions for Accurate 3D Pose Estimationcited by 122
  8. 2015Capturing Hands in Action using Discriminative Salient Points and Physics Simulationcited by 115
  9. 2017Real-time Hand Tracking under Occlusion from an Egocentric RGB-D Sensorcited by 84
  10. 2022Fine-Grained Egocentric Hand-Object Segmentation: Dataset, Model, and Applicationscited by 78
  11. 2023BOP Challenge 2022 on Detection, Segmentation and Pose Estimation of Specific Rigid Objectscited by 70
  12. 2019Photorealistic Image Synthesis for Object Instance Detectioncited by 66
  13. 2023AssemblyHands: Towards Egocentric Activity Understanding via 3D Hand Pose Estimationcited by 64
  14. 2024BOP Challenge 2023 on Detection, Segmentation and Pose Estimation of Seen and Unseen Rigid Objectscited by 60
  15. 2022Category-Level 6D Object Pose Estimation in the Wild: A Semi-Supervised Learning Approach and A New Datasetcited by 59
  16. 2016Real-time Joint Tracking of a Hand Manipulating an Object from RGB-D Inputcited by 56
  17. 2022HouseCat6D -- A Large-Scale Multi-Modal Category Level 6D Object Perception Dataset with Household Objects in Realistic Scenarioscited by 51
  18. 2022UmeTrack: Unified multi-view end-to-end hand tracking for VRcited by 48
  19. 2023HANDAL: A Dataset of Real-World Manipulable Object Categories with Pose Annotations, Affordances, and Reconstructionscited by 47
  20. 2022PhoCaL: A Multi-Modal Dataset for Category-Level Object Pose Estimation with Photometrically Challenging Objectscited by 45
  21. 20163D Hand Pose Tracking and Estimation Using Stereo Matchingcited by 35
  22. 2022In-Hand 3D Object Scanning from an RGB Sequencecited by 27
  23. 2017The 2017 Hands in the Million Challenge on 3D Hand Pose Estimationcited by 6
  24. 2018HANDS18: Methods, Techniques and Applications for Hand Observationcited by 4

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 5; 2025: 29; 2026: 12.

46 papers.

Sort:

By area:

  1. 2024WiLoR: End-to-end 3D Hand Localization and Reconstruction in-the-wildcited by 117
  2. 2025EgoVLA: Learning Vision-Language-Action Models from Egocentric Human Videoscited by 116
  3. 2024Nymeria: A Massive Collection of Multimodal Egocentric Daily Motion in the Wildcited by 109
  4. 2025HaWoR: World-Space Hand Motion Reconstruction from Egocentric Videoscited by 62
  5. 2025Open-Vocabulary Functional 3D Scene Graphs for Real-World Indoor Spacescited by 53
  6. 2026EgoVerse: An Egocentric Human Dataset for Robot Learning from Around the Worldcited by 49
  7. 2025Any6D: Model-free 6D Pose Estimation of Novel Objectscited by 47
  8. 2024GigaHands: A Massive Annotated Dataset of Bimanual Hand Activitiescited by 32
  9. 2026VGGT-Omegacited by 20
  10. 2025DeGauss: Dynamic-Static Decomposition with Gaussian Splatting for Distractor-free 3D Reconstructioncited by 16
  11. 2025Precise Action-to-Video Generation Through Visual Action Promptscited by 15
  12. 2026HumanEgo: Zero-Shot Robot Learning from Minutes of Human Egocentric Videoscited by 14
34 more
  1. 2025Reconstructing 4D Spatial Intelligence: A Surveycited by 13
  2. 2025OpenEgo: A Large-Scale Multimodal Egocentric Dataset for Dexterous Manipulationcited by 12
  3. 2025Benchmarking Egocentric Visual-Inertial SLAM at City Scalecited by 12
  4. 2025HORT: Monocular Hand-held Objects Reconstruction with Transformerscited by 11
  5. 2025Generating 6DoF Object Manipulation Trajectories from Action Description in Egocentric Visioncited by 11
  6. 2025How Do I Do That? Synthesizing 3D Hand Motion and Contacts for Everyday Interactionscited by 10
  7. 2025GoTrack: Generic 6DoF Object Pose Refinement and Trackingcited by 10
  8. 2025EgoM2P: Egocentric Multimodal Multitask Pretrainingcited by 9
  9. 2025DyTact: Capturing Dynamic Contacts in Hand-Object Manipulationcited by 8
  10. 2025Seeing in the Dark: Benchmarking Egocentric 3D Vision with the Oxford Day-and-Night Datasetcited by 8
  11. 2025Perceiving and Acting in First-Person: A Dataset and Benchmark for Egocentric Human-Object-Human Interactionscited by 7
  12. 2025DexMan: Learning Bimanual Dexterous Manipulation from Human and Generated Videoscited by 7
  13. 2026HVG-3D: Bridging Real and Simulation Domains for 3D-Conditional Hand-Object Interaction Video Synthesiscited by 7
  14. 2025Flowing from Reasoning to Motion: Learning 3D Hand Trajectory Prediction from Egocentric Human Interaction Videoscited by 6
  15. 2025ECHO: Ego-Centric modeling of Human-Object interactionscited by 4
  16. 2026Data Pyramid for Embodied Manipulation: A Surveycited by 4
  17. 2024GFreeDet: Exploiting Gaussian Splatting and Foundation Models for Model-free Unseen Object Detection in the BOP Challenge 2024cited by 3
  18. 2025HccePose(BF): Predicting Front & Back Surfaces to Construct Ultra-Dense 2D-3D Correspondences for Pose Estimationcited by 3
  19. 2025MUSE: Model-based Uncertainty-aware Similarity Estimation for zero-shot 2D Object Detection and Segmentationcited by 3
  20. 2025Beyond Blanket Masking: Examining Granularity for Privacy Protection in Images Captured by Blind and Low Vision Userscited by 2
  21. 2025HanDyVQA: A Video QA Benchmark for Fine-Grained Hand-Object Interaction Dynamicscited by 2
  22. 2026UniHand: A Unified Model for Diverse Controlled 4D Hand Motion Modelingcited by 2
  23. 2026Event6D: Event-based Novel Object 6D Pose Trackingcited by 2
  24. 2026StableHand: Quality-Aware Flow Matching for World-Space Dual-Hand Motion Estimation from Egocentric Videocited by 2
  25. 20241st Place Solution of Multiview Egocentric Hand Tracking Challenge ECCV2024cited by 1
  26. 2025MR6D: Benchmarking 6D Pose Estimation for Mobile Robotscited by 1
  27. 2025Bimanual 3D Hand Motion and Articulation Forecasting in Everyday Imagescited by 1
  28. 2026AnyHand: A Large-Scale Synthetic Dataset for RGB(-D) Hand Pose Estimationcited by 1
  29. 2025DynOPETs: A Versatile Benchmark for Dynamic Object Pose Estimation and Tracking in Moving Camera Scenarioscited by 0
  30. 2025Improving Keystep Recognition in Ego-Video via Dexterous Focuscited by 0
  31. 2025Person Identification from Egocentric Human-Object Interactions using 3D Hand Posecited by 0
  32. 2026DynaHOI: Benchmarking Hand-Object Interaction for Dynamic Targetcited by 0
  33. 2026EgoKit: Towards Unified Low-Cost Egocentric Data Collection with Heterogeneous Devicescited by 0
  34. 2026A Framework for Egocentric and Exocentric Procedural Understanding via Temporal Segmentation and Semantic Abstractioncited by 0