mlsanityData to 7 Oct 2026

3D Hand Pose Estimation in Everyday Egocentric Images

arXiv 2312.06583 ↗First posted 11 Dec 2023cs.CV cs.AI cs.LGcited by 33

Not on the map. Why?

Abstract (arXiv)

3D hand pose estimation in everyday egocentric images is challenging for several reasons: poor visual signal (occlusion from the object of interaction, low resolution & motion blur), large perspective distortion (hands are close to the camera), and lack of 3D annotations outside of controlled settings. While existing methods often use hand crops as input to focus on fine-grained visual information to deal with poor visual signal, the challenges arising from perspective distortion and lack of 3D annotations in the wild have not been systematically studied. We focus on this gap and explore the impact of different practices, i.e. crops as input, incorporating camera information, auxiliary supervision, scaling up datasets. We provide several insights that are applicable to both convolutional and transformer models leading to better performance. Based on our findings, we also present WildHands, a system for 3D hand pose estimation in everyday egocentric images. Zero-shot evaluation on 4 diverse datasets (H2O, AssemblyHands, Epic-Kitchens, Ego-Exo4D) demonstrate the effectiveness of our approach across 2D and 3D metrics, where we beat past methods by 7.4% - 66%. In system level comparisons, WildHands achieves the best 3D hand pose on ARCTIC egocentric split, outperforms FrankMocap across all metrics and HaMeR on 3 out of 6 metrics while being 10x smaller and trained on 5x less data.

Key facts. Cited by 33 papers in the map (data through 7 Oct 2026). Builds on 60 of its 78 references that are in the map. First posted on arXiv 11 Dec 2023.

Builds on

60 of 78 references.

References · 60

  1. 2017Attention Is All You Needcited by 59,932
  2. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  3. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  4. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  5. 2017Mask R-CNNcited by 7,259
  6. 2015Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shiftcited by 6,979
  7. 2016Feature Pyramid Networks for Object Detectioncited by 5,021
  8. 2021Ego4D: Around the World in 3,000 Hours of Egocentric Videocited by 1,530
  9. 2017Multi-Task Learning Using Uncertainty to Weigh Losses for Scene Geometry and Semanticscited by 1,345
  10. 2018Learning Implicit Fields for Generative Shape Modelingcited by 1,060
  11. 2017End-to-end Recovery of Human Shape and Posecited by 883
  12. 2018Scaling Egocentric Vision: The EPIC-KITCHENS Datasetcited by 789
38 more
  1. 2020Accelerating 3D Deep Learning with PyTorch3Dcited by 781
  2. 2016Monocular 3D Human Pose Estimation In The Wild Using Improved CNN Supervisioncited by 493
  3. 2023Ego-Exo4D: Understanding Skilled Human Activity from First- and Third-Person Perspectivescited by 477
  4. 2017Neural 3D Mesh Renderercited by 437
  5. 2019Soft Rasterizer: A Differentiable Renderer for Image-based 3D Reasoningcited by 420
  6. 2020GRAB: A Dataset of Whole-Body Human Grasping of Objectscited by 397
  7. 2019Learning joint reconstruction of hands and manipulated objectscited by 377
  8. 2022HOI4D: A 4D Egocentric Dataset for Category-Level Human-Object Interactioncited by 321
  9. 2021DexYCB: A Benchmark for Capturing Hand Grasping of Objectscited by 320
  10. 2018Learning Category-Specific Mesh Reconstruction from Image Collectionscited by 317
  11. 2022ARCTIC: A Dataset for Dexterous Bimanual Hand-Object Manipulationcited by 306
  12. 2023Reconstructing Hands in 3D with Transformerscited by 298
  13. 2022Assembly101: A Large-Scale Multi-View Video Dataset for Understanding Procedural Activitiescited by 284
  14. 2020Understanding Human Hands in Contact at Internet Scalecited by 281
  15. 2019FreiHAND: A Dataset for Markerless Capture of Hand Pose and Shape from Single RGB Imagescited by 271
  16. 2021H2O: Two Hands Manipulating Objects for First Person Interaction Recognitioncited by 251
  17. 2020InterHand2.6M: A Dataset and Baseline for 3D Interacting Hand Pose Estimation from a Single RGB Imagecited by 244
  18. 2021FIERY: Future Instance Prediction in Bird's-Eye View from Surround Monocular Camerascited by 239
  19. 2017Learning to Estimate 3D Hand Pose from Single RGB Imagescited by 234
  20. 2017Hand Keypoint Detection in Single Images using Multiview Bootstrappingcited by 231
  21. 2017First-Person Hand Action Benchmark with RGB-D Videos and 3D Hand Pose Annotationscited by 229
  22. 2020The EPIC-KITCHENS Dataset: Collection, Challenges and Baselinescited by 221
  23. 2020Grasping Field: Learning Implicit Representations for Human Graspscited by 178
  24. 2017Multi-view Supervision for Single-view Reconstruction via Differentiable Ray Consistencycited by 177
  25. 2022Omni3D: A Large Benchmark and Model for 3D Object Detection in the Wildcited by 159
  26. 2020Reconstructing Hand-Object Interactions in the Wildcited by 143
  27. 2020Leveraging Photometric Consistency over Time for Sparsely Supervised Hand-Object Reconstructioncited by 139
  28. 2022EPIC-KITCHENS VISOR Benchmark: VIdeo Segmentations and Object Relationscited by 139
  29. 2022OakInk: A Large-scale Knowledge Repository for Understanding Hand-Object Interactioncited by 129
  30. 2019End-to-end Hand Mesh Recovery from a Monocular RGB Imagecited by 124
  31. 2021Keypoint Transformer: Solving Joint Identification in Challenging Hands and Object Interactions for Accurate 3D Pose Estimationcited by 122
  32. 2023Towards Zero-Shot Scale-Aware Monocular Depth Estimationcited by 100
  33. 2022HandOccNet: Occlusion-Robust 3D Hand Mesh Estimation Networkcited by 95
  34. 2022What's in your hands? 3D Reconstruction of Generic Objects in Handscited by 90
  35. 2019CAM-Convs: Camera-Aware Multi-Scale Convolutions for Single-View Depthcited by 75
  36. 2023GTA: A Geometry-Aware Attention Mechanism for Multi-View Transformerscited by 75
  37. 2020FrankMocap: Fast Monocular 3D Hand and Body Motion Capture by Regression and Integrationcited by 72
  38. 2023AssemblyHands: Towards Egocentric Activity Understanding via 3D Hand Pose Estimationcited by 64

50 most cited of 60 All 60 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 1; 2024: 2; 2025: 11; 2026: 19.

33 papers.

Sort:

By area:

  1. 2024Benchmarks and Challenges in Pose Estimation for Egocentric Hand Interactions with Objectscited by 35
  2. 2024Reconstructing Hand-Held Objects in 3D from Images and Videoscited by 23
  3. 2025Challenges and Trends in Egocentric Vision: A Surveycited by 20
  4. 2025PlayerOne: Egocentric World Simulatorcited by 20
  5. 2025The Invisible EgoHand: 3D Hand Forecasting through EgoBody Pose Estimationcited by 16
  6. 2023Get a Grip: Reconstructing Hand-Object Stable Grasps in Egocentric Videoscited by 10
  7. 2025How Do I Do That? Synthesizing 3D Hand Motion and Contacts for Everyday Interactionscited by 10
  8. 2026EgoGrasp: World-Space Hand-Object Interaction Estimation from Egocentric Videoscited by 7
  9. 2025EventEgoHands: Event-based Egocentric 3D Hand Mesh Reconstructioncited by 5
  10. 2026SHOW3D: Capturing Scenes of 3D Hands and Objects in the Wildcited by 4
  11. 2026The Surprising Effectiveness of Video Diffusion Models for Hand Motion Reconstructioncited by 4
  12. 2026EgoForce: Forearm-Guided Camera-Space 3D Hand Pose from a Monocular Egocentric Cameracited by 3
21 more
  1. 2025Affordance-Guided Diffusion Prior for 3D Hand Reconstructioncited by 2
  2. 2025Reconstructing Objects along Hand Interaction Timelines in Egocentric Videocited by 2
  3. 2026Towards Egocentric 3D Hand Pose Estimation in Unseen Domainscited by 2
  4. 2026EgoHandICL: Egocentric 3D Hand Reconstruction with In-Context Learningcited by 2
  5. 2026Detecting Precise Hand Touch Moments in Egocentric Videocited by 2
  6. 2026StableHand: Quality-Aware Flow Matching for World-Space Dual-Hand Motion Estimation from Egocentric Videocited by 2
  7. 2026MINT: A Unified Model for World-Space Camera and Hand Motion Estimation from Scalable Egocentric Pipeline Supervisioncited by 2
  8. 2025Bimanual 3D Hand Motion and Articulation Forecasting in Everyday Imagescited by 1
  9. 2026DeltaDorsal: Enhancing Hand Pose Estimation with Dorsal Features in Egocentric Viewscited by 1
  10. 2026ACE-Ego-Hand: Repurposing Video Diffusion Models for Occlusion-Robust Egocentric 3D Hand Motion Recoverycited by 1
  11. 2025Learning Transformation-Isomorphic Latent Space for Accurate Hand Pose Estimationcited by 0
  12. 2025EgoLoc: A Generalizable Solution for Temporal Interaction Localization in Egocentric Videoscited by 0
  13. 2025DF-Mamba: Deformable State Space Modeling for 3D Hand Pose Estimation in Interactionscited by 0
  14. 2026AirGlove: Exploring Egocentric 3D Hand Tracking and Appearance Generalization for Sensing Glovescited by 0
  15. 2026Impact of Hand Impairment and Occlusions on Hand Pose Estimation Accuracy in Augmented Reality Applicationscited by 0
  16. 2026Exo2EgoPose: Leveraging Exocentric Demonstrations for Vision-Language guided Egocentric 3D Hand Pose Forecastingcited by 0
  17. 2026Hand-Object Interaction in the Age of Large Foundation Models:Reconstruction, Generation, and Embodied Transfercited by 0
  18. 2026ACE-Data-0: Human-Centric Ambient Capture as Embodied Data Enginecited by 0
  19. 2026TransHands: Repurposing Human Pose Encoders as Hand Pose Encoderscited by 0
  20. 2026EventEgoHands++: Event-based Egocentric 3D Hand Mesh Reconstruction with Real Datasetcited by 0
  21. 2026InfiniHand: Streaming World-Space Hand Motion Estimation from Egocentric Videocited by 0