mlsanityData to 7 Oct 2026

HO-Cap: A Capture System and Dataset for 3D Reconstruction and Pose Tracking of Hand-Object Interaction

arXiv 2406.06843 ↗First posted 10 Jun 2024cs.CVcited by 25

Not on the map. Why?

Abstract (arXiv)

We introduce a data capture system and a new dataset, HO-Cap, for 3D reconstruction and pose tracking of hands and objects in videos. The system leverages multiple RGBD cameras and a HoloLens headset for data collection, avoiding the use of expensive 3D scanners or mocap systems. We propose a semi-automatic method for annotating the shape and pose of hands and objects in the collected videos, significantly reducing the annotation time compared to manual labeling. With this system, we captured a video dataset of humans interacting with objects to perform various tasks, including simple pick-and-place actions, handovers between hands, and using objects according to their affordance, which can serve as human demonstrations for research in embodied AI and robot manipulation. Our data capture setup and annotation framework will be available for the community to use in reconstructing 3D shapes of objects and human hands and tracking their poses in videos.

Key facts. Cited by 25 papers in the map (data through 7 Oct 2026). Builds on 43 of its 59 references that are in the map. First posted on arXiv 10 Jun 2024.

Builds on

43 of 59 references.

References · 43

  1. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  2. 2014Microsoft COCO: Common Objects in Contextcited by 17,498
  3. 2023Segment Anythingcited by 8,158
  4. 2023DINOv2: Learning Robust Visual Features without Supervisioncited by 7,764
  5. 2023Grounding DINO: Marrying DINO with Grounded Pre-Training for Open-Set Object Detectioncited by 3,274
  6. 2021LoFTR: Detector-Free Local Feature Matching with Transformerscited by 974
  7. 2017PoseCNN: A Convolutional Neural Network for 6D Object Pose Estimation in Cluttered Scenescited by 768
  8. 2023DETRs Beat YOLOs on Real-time Object Detectioncited by 526
  9. 2022XMem: Long-Term Video Object Segmentation with an Atkinson-Shiffrin Memory Modelcited by 436
  10. 2023FoundationPose: Unified 6D Pose Estimation and Tracking of Novel Objectscited by 411
  11. 2020GRAB: A Dataset of Whole-Body Human Grasping of Objectscited by 397
  12. 2019Learning joint reconstruction of hands and manipulated objectscited by 377
31 more
  1. 2022HOI4D: A 4D Egocentric Dataset for Category-Level Human-Object Interactioncited by 321
  2. 2021DexYCB: A Benchmark for Capturing Hand Grasping of Objectscited by 320
  3. 2022ARCTIC: A Dataset for Dexterous Bimanual Hand-Object Manipulationcited by 306
  4. 2023Reconstructing Hands in 3D with Transformerscited by 298
  5. 2020Understanding Human Hands in Contact at Internet Scalecited by 281
  6. 2021H2O: Two Hands Manipulating Objects for First Person Interaction Recognitioncited by 251
  7. 2020InterHand2.6M: A Dataset and Baseline for 3D Interacting Hand Pose Estimation from a Single RGB Imagecited by 244
  8. 2017First-Person Hand Action Benchmark with RGB-D Videos and 3D Hand Pose Annotationscited by 229
  9. 2022MegaPose: 6D Pose Estimation of Novel Objects via Render & Comparecited by 198
  10. 2020ContactPose: A Dataset of Grasps with Object Contact and Hand Posecited by 180
  11. 2020MediaPipe Hands: On-device Real-time Hand Trackingcited by 157
  12. 2023BundleSDF: Neural 6-DoF Tracking and 3D Reconstruction of Unknown Objectscited by 144
  13. 2022EPIC-KITCHENS VISOR Benchmark: VIdeo Segmentations and Object Relationscited by 139
  14. 2022OakInk: A Large-scale Knowledge Repository for Understanding Hand-Object Interactioncited by 129
  15. 2022DiffPose: Toward More Reliable 3D Pose Estimationcited by 99
  16. 2015Benchmarking in Manipulation Research: The YCB Object and Model Set and Benchmarking Protocolscited by 98
  17. 2019PoseRBPF: A Rao-Blackwellized Particle Filter for 6D Object Pose Trackingcited by 91
  18. 2021BundleTrack: 6D Pose Tracking for Novel Objects without Instance or Category-Level 3D Modelscited by 87
  19. 2023CNOS: A Strong Baseline for CAD-based Novel Object Segmentationcited by 86
  20. 2024OAKINK2: A Dataset of Bimanual Hands-Object Manipulation in Complex Task Completioncited by 78
  21. 2021Real-time Pose and Shape Reconstruction of Two Interacting Hands With a Single Depth Cameracited by 70
  22. 2019A2J: Anchor-to-Joint Regression Network for 3D Articulated Pose Estimation from a Single Depth Imagecited by 56
  23. 2023gSDF: Geometry-Driven Signed Distance Functions for 3D Hand-Object Reconstructioncited by 49
  24. 2023HANDAL: A Dataset of Real-World Manipulable Object Categories with Pose Annotations, Affordances, and Reconstructionscited by 47
  25. 2023AffordPose: A Large-scale Dataset of Hand-Object Interactions with Affordance-driven Hand Posecited by 47
  26. 2023Bringing Inputs to Shared Domains for 3D Interacting Hands Recovery in the Wildcited by 36
  27. 2023TTA-COPE: Test-Time Adaptation for Category-Level Object Pose Estimationcited by 32
  28. 2022Consistent 3D Hand Reconstruction in Video via self-supervised Learningcited by 28
  29. 2023SHOWMe: Benchmarking Object-agnostic Hand-Object 3D Reconstructioncited by 16
  30. 2022CRT-6D: Fast 6D Object Pose Estimation with Cascaded Refinement Transformerscited by 15
  31. 2023A2J-Transformer: Anchor-to-Joint Transformer Network for 3D Interacting Hand Pose Estimation from a Single RGB Imagecited by 15

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 1; 2025: 8; 2026: 16.

25 papers.

Sort:

By area:

  1. 2024HOT3D: Hand and Object Tracking in 3D from Egocentric Multi-View Videoscited by 122
  2. 2025villa-X: Enhancing Latent Action Modeling in Vision-Language-Action Modelscited by 58
  3. 2025MonoFusion: Sparse-View 4D Reconstruction via Monocular Fusioncited by 18
  4. 2025OpenEgo: A Large-Scale Multimodal Egocentric Dataset for Dexterous Manipulationcited by 12
  5. 2025Co-Evolving Latent Action World Modelscited by 12
  6. 2025HORT: Monocular Hand-held Objects Reconstruction with Transformerscited by 11
  7. 2025Learning 3D-Gaussian Simulators from RGB Videoscited by 8
  8. 2025DyTact: Capturing Dynamic Contacts in Hand-Object Manipulationcited by 8
  9. 2026FunRec: Reconstructing Functional 3D Scenes from Egocentric Interaction Videoscited by 7
  10. 2026Learning Additively Compositional Latent Actions for Embodied AIcited by 6
  11. 2026SHOW3D: Capturing Scenes of 3D Hands and Objects in the Wildcited by 4
  12. 2026Human-Centric Transferable Tactile Pre-Training for Dexterous Robotic Manipulationcited by 4
13 more
  1. 2026HandEdit: A Unified Benchmark for Egocentric Human-to-Robot Dexterous Hand Image Editingcited by 4
  2. 2026Glove2Hand: Synthesizing Natural Hand-Object Interaction from Multi-Modal Sensing Glovescited by 3
  3. 2026PAM: A Pose-Appearance-Motion Engine for Sim-to-Real HOI Video Generationcited by 3
  4. 2026EgoTL: Egocentric Think-Aloud Chains for Long-Horizon Taskscited by 3
  5. 2025Reconstructing Objects along Hand Interaction Timelines in Egocentric Videocited by 2
  6. 2026Event6D: Event-based Novel Object 6D Pose Trackingcited by 2
  7. 2026HRDexDB: A 4D Dexterous Grasping Dataset Across Human and Multiple Robot Embodimentscited by 2
  8. 2026Hand Visibility Detector: Per-Keypoint Visibility Estimation for Handscited by 2
  9. 2026EgoTac: In-the-wild Tactile Prediction from Egocentric Visioncited by 2
  10. 2026AnyHand: A Large-Scale Synthetic Dataset for RGB(-D) Hand Pose Estimationcited by 1
  11. 2026EgoKit: Towards Unified Low-Cost Egocentric Data Collection with Heterogeneous Devicescited by 0
  12. 2026Hand-Object Interaction in the Age of Large Foundation Models:Reconstruction, Generation, and Embodied Transfercited by 0
  13. 2026RoboTok: A Scalable Data Engine for Internet Demonstration Video Retrieval and Dexterous Manipulation Learningcited by 0