mlsanityData to 7 Oct 2026

AssemblyHands: Towards Egocentric Activity Understanding via 3D Hand Pose Estimation

arXiv 2304.12301 ↗First posted 24 Apr 2023cs.CVcited by 64

Not on the map. Why?

Abstract (arXiv)

We present AssemblyHands, a large-scale benchmark dataset with accurate 3D hand pose annotations, to facilitate the study of egocentric activities with challenging hand-object interactions. The dataset includes synchronized egocentric and exocentric images sampled from the recent Assembly101 dataset, in which participants assemble and disassemble take-apart toys. To obtain high-quality 3D hand pose annotations for the egocentric images, we develop an efficient pipeline, where we use an initial set of manual annotations to train a model to automatically annotate a much larger dataset. Our annotation model uses multi-view feature fusion and an iterative refinement scheme, and achieves an average keypoint error of 4.20 mm, which is 85% lower than the error of the original annotations in Assembly101. AssemblyHands provides 3.0M annotated images, including 490K egocentric images, making it the largest existing benchmark dataset for egocentric 3D hand pose estimation. Using this data, we develop a strong single-view baseline of 3D hand pose estimation from egocentric images. Furthermore, we design a novel action classification task to evaluate predicted 3D hand poses. Our study shows that having higher-quality hand poses directly improves the ability to recognize actions.

Key facts. Cited by 64 papers in the map (data through 7 Oct 2026). Builds on 30 of its 40 references that are in the map. First posted on arXiv 24 Apr 2023.

Builds on

30 of 40 references.

References · 30

  1. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  2. 2019EfficientNet: Rethinking Model Scaling for Convolutional Neural Networkscited by 6,415
  3. 2021Ego4D: Around the World in 3,000 Hours of Egocentric Videocited by 1,530
  4. 2018OpenPose: Realtime Multi-Person 2D Pose Estimation using Part Affinity Fieldscited by 1,161
  5. 2017Spatio-Temporal Graph Convolutional Networks: A Deep Learning Framework for Traffic Forecastingcited by 1,125
  6. 2018TSM: Temporal Shift Module for Efficient Video Understandingcited by 852
  7. 2016NTU RGB+D: A Large Scale Dataset for 3D Human Activity Analysiscited by 683
  8. 2019Learning joint reconstruction of hands and manipulated objectscited by 377
  9. 2020Disentangling and Unifying Graph Convolutions for Skeleton-Based Action Recognitioncited by 339
  10. 2021DexYCB: A Benchmark for Capturing Hand Grasping of Objectscited by 320
  11. 2022Assembly101: A Large-Scale Multi-View Video Dataset for Understanding Procedural Activitiescited by 284
  12. 2019FreiHAND: A Dataset for Markerless Capture of Hand Pose and Shape from Single RGB Imagescited by 271
18 more
  1. 2021H2O: Two Hands Manipulating Objects for First Person Interaction Recognitioncited by 251
  2. 2020InterHand2.6M: A Dataset and Baseline for 3D Interacting Hand Pose Estimation from a Single RGB Imagecited by 244
  3. 2017Learning to Estimate 3D Hand Pose from Single RGB Imagescited by 234
  4. 2017Hand Keypoint Detection in Single Images using Multiview Bootstrappingcited by 231
  5. 2017First-Person Hand Action Benchmark with RGB-D Videos and 3D Hand Pose Annotationscited by 229
  6. 2018Human Action Recognition and Prediction: A Surveycited by 226
  7. 2022Embodied Hands: Modeling and Capturing Hands and Bodies Togethercited by 179
  8. 2019Learnable Triangulation of Human Posecited by 160
  9. 2018Hand Pose Estimation via Latent 2.5D Heatmap Regressioncited by 111
  10. 2017V2V-PoseNet: Voxel-to-Voxel Prediction Network for Accurate 3D Hand and Human Pose Estimation from a Single Depth Mapcited by 102
  11. 2017Real-time Hand Tracking under Occlusion from an Egocentric RGB-D Sensorcited by 84
  12. 2015P-CNN: Pose-based CNN Features for Action Recognitioncited by 58
  13. 2016Real-time Joint Tracking of a Hand Manipulating an Object from RGB-D Inputcited by 56
  14. 2022UmeTrack: Unified multi-view end-to-end hand tracking for VRcited by 48
  15. 2021Generalizable Human Pose Triangulationcited by 24
  16. 2022Efficient Annotation and Learning for 3D Hand Pose Estimation: A Surveycited by 17
  17. 2016Pose for Action - Action for Posecited by 16
  18. 2022Domain Adaptive Hand Keypoint and Pixel Localization in the Wildcited by 14

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 7; 2024: 19; 2025: 17; 2026: 21.

64 papers.

Sort:

By area:

  1. 2023Ego-Exo4D: Understanding Skilled Human Activity from First- and Third-Person Perspectivescited by 477
  2. 2024HOT3D: Hand and Object Tracking in 3D from Egocentric Multi-View Videoscited by 122
  3. 2023An Outlook into the Future of Egocentric Visioncited by 84
  4. 2024Introducing HOT3D: An Egocentric Dataset for 3D Hand and Object Trackingcited by 46
  5. 2023A Dataset of Relighted 3D Interacting Handscited by 35
  6. 2024Benchmarks and Challenges in Pose Estimation for Egocentric Hand Interactions with Objectscited by 35
  7. 20233D Hand Pose Estimation in Everyday Egocentric Imagescited by 33
  8. 2024GigaHands: A Massive Annotated Dataset of Bimanual Hand Activitiescited by 32
  9. 2025Challenges and Trends in Egocentric Vision: A Surveycited by 20
  10. 2024EgoPressure: A Dataset for Hand Pressure and Pose Estimation in Egocentric Visioncited by 18
  11. 2024On the Utility of 3D Hand Poses for Action Recognitioncited by 17
  12. 2024EAGLE: Egocentric AGgregated Language-video Enginecited by 16
38 more
  1. 2026AnyView: Synthesizing Any Novel View in Dynamic Scenescited by 12
  2. 2023Exo2EgoDVC: Dense Video Captioning of Egocentric Procedural Activities Using Web Instructional Videoscited by 11
  3. 2024AttentionHand: Text-driven Controllable Hand Image Generation for 3D Hand Reconstruction in the Wildcited by 10
  4. 2024PEAR: Phrase-Based Hand-Object Interaction Anticipationcited by 10
  5. 2025EgoM2P: Egocentric Multimodal Multitask Pretrainingcited by 9
  6. 2024EMAG: Ego-motion Aware and Generalizable 2D Hand Forecasting from Egocentric Videoscited by 8
  7. 2024FoundHand: Large-Scale Domain-Specific Learning for Controllable Hand Image Generationcited by 8
  8. 2025DyTact: Capturing Dynamic Contacts in Hand-Object Manipulationcited by 8
  9. 2025Bridging Perspectives: A Survey on Cross-view Collaborative Intelligence with Egocentric-Exocentric Visioncited by 8
  10. 2025Kinematify: Open-Vocabulary Synthesis of High-DoF Articulated Objectscited by 8
  11. 2025SiMHand: Mining Similar Hands for Large-Scale 3D Hand Pose Pre-trainingcited by 7
  12. 2026EgoGrasp: World-Space Hand-Object Interaction Estimation from Egocentric Videoscited by 7
  13. 2025EventEgoHands: Event-based Egocentric 3D Hand Mesh Reconstructioncited by 5
  14. 2025egoEMOTION: Egocentric Vision and Physiological Signals for Emotion and Personality Recognition in Real-World Taskscited by 5
  15. 2024ThermoHands: A Benchmark for 3D Hand Pose Estimation from Egocentric Thermal Imagescited by 4
  16. 2026Data Pyramid for Embodied Manipulation: A Surveycited by 4
  17. 20231st Place Solution of Egocentric 3D Hand Pose Estimation Challenge 2023 Technical Report:A Concise Pipeline for Egocentric Hand Pose Reconstructioncited by 3
  18. 2023Generative Hierarchical Temporal Transformer for Hand Pose and Action Modelingcited by 3
  19. 2024Benchmarking 2D Egocentric Hand Pose Datasetscited by 3
  20. 2024CHASE: Learning Convex Hull Adaptive Shift for Skeleton-based Multi-Entity Action Recognitioncited by 3
  21. 2025EgoEvGesture: Gesture Recognition Based on Egocentric Event Cameracited by 3
  22. 2025Egocentric Human-Object Interaction Detection: A New Benchmark and Methodcited by 3
  23. 2024Pre-Training for 3D Hand Pose Estimation with Contrastive Learning on Large-Scale Hand Images in the Wildcited by 2
  24. 2024ConvMixFormer- A Resource-efficient Convolution Mixer for Transformer-based Dynamic Hand Gesture Recognitioncited by 2
  25. 2025EgoBrain: Synergizing Minds and Eyes For Human Action Understandingcited by 2
  26. 2025Bidirectional Action Sequence Learning for Long-term Action Anticipation with Large Language Modelscited by 2
  27. 2025Generative Modeling of Shape-Dependent Self-Contact Human Posescited by 2
  28. 2025Affordance-Guided Diffusion Prior for 3D Hand Reconstructioncited by 2
  29. 2025Gaze Beyond the Frame: Forecasting Egocentric 3D Visual Spancited by 2
  30. 2026Towards Egocentric 3D Hand Pose Estimation in Unseen Domainscited by 2
  31. 2026EgoEV-HandPose: Egocentric 3D Hand Pose Estimation and Gesture Recognition with Stereo Event Camerascited by 2
  32. 2026YUBI: Yielding Universal Bidigital Interface for Bimanual Dexterous Manipulation at Scalecited by 2
  33. 2026HandsOnWorld: Unconstrained Egocentric Video Generation with Camera-Disentangled Hand Controlcited by 2
  34. 2026Hand Visibility Detector: Per-Keypoint Visibility Estimation for Handscited by 2
  35. 2024ADL4D: Towards A Contextually Rich Dataset for 4D Activities of Daily Livingcited by 1
  36. 2024SHARP: Segmentation of Hands and Arms by Range using Pseudo-Depth for Enhanced Egocentric 3D Hand Pose Estimation and Action Recognitioncited by 1
  37. 20241st Place Solution of Multiview Egocentric Hand Tracking Challenge ECCV2024cited by 1
  38. 2025AssemblyHands-X: Modeling 3D Hand-Body Coordination for Understanding Bimanual Human Activitiescited by 1

50 most cited of 64 All 64 (JSON)