# 3D Hand Pose Estimation in Everyday Egocentric Images

arXiv: https://arxiv.org/abs/2312.06583 · HTML: https://mlsanity.com/paper/2312.06583/

## Key facts

- Cited by 34 papers in the map (data through 14 Oct 2026).
- Builds on 60 of its 78 references that are in the map.
- First posted on arXiv 11 Dec 2023.

## Builds on (most cited first, 30 of 60)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [Mask R-CNN](https://mlsanity.com/paper/1703.06870/) (2017)
- [Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift](https://mlsanity.com/paper/1502.03167/) (2015)
- [Feature Pyramid Networks for Object Detection](https://mlsanity.com/paper/1612.03144/) (2016)
- [Ego4D: Around the World in 3,000 Hours of Egocentric Video](https://mlsanity.com/paper/2110.07058/) (2021)
- [Multi-Task Learning Using Uncertainty to Weigh Losses for Scene Geometry and Semantics](https://mlsanity.com/paper/1705.07115/) (2017)
- [Learning Implicit Fields for Generative Shape Modeling](https://mlsanity.com/paper/1812.02822/) (2018)
- [End-to-end Recovery of Human Shape and Pose](https://mlsanity.com/paper/1712.06584/) (2017)
- [Scaling Egocentric Vision: The EPIC-KITCHENS Dataset](https://mlsanity.com/paper/1804.02748/) (2018)
- [Accelerating 3D Deep Learning with PyTorch3D](https://mlsanity.com/paper/2007.08501/) (2020)
- [Monocular 3D Human Pose Estimation In The Wild Using Improved CNN Supervision](https://mlsanity.com/paper/1611.09813/) (2016)
- [Ego-Exo4D: Understanding Skilled Human Activity from First- and Third-Person Perspectives](https://mlsanity.com/paper/2311.18259/) (2023)
- [Neural 3D Mesh Renderer](https://mlsanity.com/paper/1711.07566/) (2017)
- [Soft Rasterizer: A Differentiable Renderer for Image-based 3D Reasoning](https://mlsanity.com/paper/1904.01786/) (2019)
- [GRAB: A Dataset of Whole-Body Human Grasping of Objects](https://mlsanity.com/paper/2008.11200/) (2020)
- [Learning joint reconstruction of hands and manipulated objects](https://mlsanity.com/paper/1904.05767/) (2019)
- [HOI4D: A 4D Egocentric Dataset for Category-Level Human-Object Interaction](https://mlsanity.com/paper/2203.01577/) (2022)
- [DexYCB: A Benchmark for Capturing Hand Grasping of Objects](https://mlsanity.com/paper/2104.04631/) (2021)
- [Learning Category-Specific Mesh Reconstruction from Image Collections](https://mlsanity.com/paper/1803.07549/) (2018)
- [ARCTIC: A Dataset for Dexterous Bimanual Hand-Object Manipulation](https://mlsanity.com/paper/2204.13662/) (2022)
- [Reconstructing Hands in 3D with Transformers](https://mlsanity.com/paper/2312.05251/) (2023)
- [Assembly101: A Large-Scale Multi-View Video Dataset for Understanding Procedural Activities](https://mlsanity.com/paper/2203.14712/) (2022)
- [Understanding Human Hands in Contact at Internet Scale](https://mlsanity.com/paper/2006.06669/) (2020)
- [FreiHAND: A Dataset for Markerless Capture of Hand Pose and Shape from Single RGB Images](https://mlsanity.com/paper/1909.04349/) (2019)
- [H2O: Two Hands Manipulating Objects for First Person Interaction Recognition](https://mlsanity.com/paper/2104.11181/) (2021)
- [InterHand2.6M: A Dataset and Baseline for 3D Interacting Hand Pose Estimation from a Single RGB Image](https://mlsanity.com/paper/2008.09309/) (2020)
- [FIERY: Future Instance Prediction in Bird's-Eye View from Surround Monocular Cameras](https://mlsanity.com/paper/2104.10490/) (2021)

## Built on by (most cited first, 30 of 34)

- [Benchmarks and Challenges in Pose Estimation for Egocentric Hand Interactions with Objects](https://mlsanity.com/paper/2403.16428/) (2024)
- [Reconstructing Hand-Held Objects in 3D from Images and Videos](https://mlsanity.com/paper/2404.06507/) (2024)
- [Challenges and Trends in Egocentric Vision: A Survey](https://mlsanity.com/paper/2503.15275/) (2025)
- [PlayerOne: Egocentric World Simulator](https://mlsanity.com/paper/2506.09995/) (2025)
- [The Invisible EgoHand: 3D Hand Forecasting through EgoBody Pose Estimation](https://mlsanity.com/paper/2504.08654/) (2025)
- [Get a Grip: Reconstructing Hand-Object Stable Grasps in Egocentric Videos](https://mlsanity.com/paper/2312.15719/) (2023)
- [How Do I Do That? Synthesizing 3D Hand Motion and Contacts for Everyday Interactions](https://mlsanity.com/paper/2504.12284/) (2025)
- [EgoGrasp: World-Space Hand-Object Interaction Estimation from Egocentric Videos](https://mlsanity.com/paper/2601.01050/) (2026)
- [EventEgoHands: Event-based Egocentric 3D Hand Mesh Reconstruction](https://mlsanity.com/paper/2505.19169/) (2025)
- [The Surprising Effectiveness of Video Diffusion Models for Hand Motion Reconstruction](https://mlsanity.com/paper/2606.30308/) (2026)
- [SHOW3D: Capturing Scenes of 3D Hands and Objects in the Wild](https://mlsanity.com/paper/2603.28760/) (2026)
- [Reconstructing Objects along Hand Interaction Timelines in Egocentric Video](https://mlsanity.com/paper/2512.07394/) (2025)
- [Towards Egocentric 3D Hand Pose Estimation in Unseen Domains](https://mlsanity.com/paper/2601.06537/) (2026)
- [EgoHandICL: Egocentric 3D Hand Reconstruction with In-Context Learning](https://mlsanity.com/paper/2601.19850/) (2026)
- [EgoForce: Forearm-Guided Camera-Space 3D Hand Pose from a Monocular Egocentric Camera](https://mlsanity.com/paper/2605.12498/) (2026)
- [ACE-Data-0: Human-Centric Ambient Capture as Embodied Data Engine](https://mlsanity.com/paper/2607.28625/) (2026)
- [Affordance-Guided Diffusion Prior for 3D Hand Reconstruction](https://mlsanity.com/paper/2510.00506/) (2025)
- [Detecting Precise Hand Touch Moments in Egocentric Video](https://mlsanity.com/paper/2604.12343/) (2026)
- [StableHand: Quality-Aware Flow Matching for World-Space Dual-Hand Motion Estimation from Egocentric Video](https://mlsanity.com/paper/2605.18553/) (2026)
- [ACE-Ego-Hand: Repurposing Video Diffusion Models for Occlusion-Robust Egocentric 3D Hand Motion Recovery](https://mlsanity.com/paper/2608.20308/) (2026)
- [MINT: A Unified Model for World-Space Camera and Hand Motion Estimation from Scalable Egocentric Pipeline Supervision](https://mlsanity.com/paper/2609.04958/) (2026)
- [EgoLoc: A Generalizable Solution for Temporal Interaction Localization in Egocentric Videos](https://mlsanity.com/paper/2508.12349/) (2025)
- [Bimanual 3D Hand Motion and Articulation Forecasting in Everyday Images](https://mlsanity.com/paper/2510.06145/) (2025)
- [DeltaDorsal: Enhancing Hand Pose Estimation with Dorsal Features in Egocentric Views](https://mlsanity.com/paper/2601.15516/) (2026)
- [Hand-Object Interaction in the Age of Large Foundation Models:Reconstruction, Generation, and Embodied Transfer](https://mlsanity.com/paper/2607.28394/) (2026)
- [Learning Transformation-Isomorphic Latent Space for Accurate Hand Pose Estimation](https://mlsanity.com/paper/2502.12535/) (2025)
- [DF-Mamba: Deformable State Space Modeling for 3D Hand Pose Estimation in Interactions](https://mlsanity.com/paper/2512.02727/) (2025)
- [AirGlove: Exploring Egocentric 3D Hand Tracking and Appearance Generalization for Sensing Gloves](https://mlsanity.com/paper/2602.05159/) (2026)
- [Impact of Hand Impairment and Occlusions on Hand Pose Estimation Accuracy in Augmented Reality Applications](https://mlsanity.com/paper/2606.17427/) (2026)
- [Exo2EgoPose: Leveraging Exocentric Demonstrations for Vision-Language guided Egocentric 3D Hand Pose Forecasting](https://mlsanity.com/paper/2607.15890/) (2026)

## Abstract (arXiv)

3D hand pose estimation in everyday egocentric images is challenging for several reasons: poor visual signal (occlusion from the object of interaction, low resolution & motion blur), large perspective distortion (hands are close to the camera), and lack of 3D annotations outside of controlled settings. While existing methods often use hand crops as input to focus on fine-grained visual information to deal with poor visual signal, the challenges arising from perspective distortion and lack of 3D annotations in the wild have not been systematically studied. We focus on this gap and explore the impact of different practices, i.e. crops as input, incorporating camera information, auxiliary supervision, scaling up datasets. We provide several insights that are applicable to both convolutional and transformer models leading to better performance. Based on our findings, we also present WildHands, a system for 3D hand pose estimation in everyday egocentric images. Zero-shot evaluation on 4 diverse datasets (H2O, AssemblyHands, Epic-Kitchens, Ego-Exo4D) demonstrate the effectiveness of our approach across 2D and 3D metrics, where we beat past methods by 7.4% - 66%. In system level comparisons, WildHands achieves the best 3D hand pose on ARCTIC egocentric split, outperforms FrankMocap across all metrics and HaMeR on 3 out of 6 metrics while being 10x smaller and trained on 5x less data.