Towards in-the-wild Egocentric 3D Hand-Object Pose Estimation
Not on the map. Why?
Abstract (arXiv)
Estimating accurate 3D hand-object pose from in-the-wild egocentric RGB remains challenging due to severe occlusions and ambiguous contact. Existing learning-based methods often struggle to generalise to in-the-wild scenes and are limited by the scarcity of supervision. We address these issues with two contributions. First, we introduce EPIC-Contact, an in-the-wild egocentric dataset of 2.3K clips (62.3K frames) with dense, bijective 3D hand-object contact correspondences and posed meshes. Second, we propose HOPformer, an end-to-end transformer that jointly predicts bi-manual hand and object pose in a single forward pass. A cross-attention decoder conditions object features on hand priors, producing robust pose estimation. We test HOPformer on the in-lab 3D dataset, ARCTIC, as well as our newly introduced EPIC-Contact dataset. HOPformer reaches 82.4% success rate on ARCTIC (+6.2 pts over current SOTA). On EPIC-Contact, it nearly doubles the success rate while reducing contact deviation by 75%. EPIC-Contact, HOPformer code and checkpoints are released: https://sid2697.github.io/epic-contact.
Key facts. Cited by 2 papers in the map (data through 14 Oct 2026). Builds on 69 of its 76 references that are in the map. First posted on arXiv 29 Jun 2026.
Builds on
69 of 76 references.
References · 69
- 2015Deep Residual Learning for Image Recognitioncited by 52,096
- 2023DINOv2: Learning Robust Visual Features without Supervisioncited by 7,974
- 2020End-to-End Object Detection with Transformerscited by 7,414
- 2025Gemini 2.5: Pushing the Frontier with Advanced Reasoning, Multimodality, Long Context, and Next Generation Agentic Capabilitiescited by 3,796
- 2018On the Continuity of Rotation Representations in Neural Networkscited by 1,214
- 2021Scaling Vision Transformerscited by 999
- 2017End-to-end Recovery of Human Shape and Posecited by 885
- 2020Rescaling Egocentric Visioncited by 540
- 2020End-to-End Human Pose and Mesh Reconstruction with Transformerscited by 418
- 2020GRAB: A Dataset of Whole-Body Human Grasping of Objectscited by 405
- 2021PARE: Part Attention Regressor for 3D Human Body Estimationcited by 337
- 2022HOI4D: A 4D Egocentric Dataset for Category-Level Human-Object Interactioncited by 327
38 more
- 2021DexYCB: A Benchmark for Capturing Hand Grasping of Objectscited by 325
- 2022ARCTIC: A Dataset for Dexterous Bimanual Hand-Object Manipulationcited by 312
- 2023Reconstructing Hands in 3D with Transformerscited by 306
- 2019FreiHAND: A Dataset for Markerless Capture of Hand Pose and Shape from Single RGB Imagescited by 271
- 2021H2O: Two Hands Manipulating Objects for First Person Interaction Recognitioncited by 254
- 2025SAM 3D: 3Dfy Anything in Imagescited by 247
- 2020InterHand2.6M: A Dataset and Baseline for 3D Interacting Hand Pose Estimation from a Single RGB Imagecited by 245
- 2017Hand Keypoint Detection in Single Images using Multiview Bootstrappingcited by 231
- 2017First-Person Hand Action Benchmark with RGB-D Videos and 3D Hand Pose Annotationscited by 230
- 20193D Hand Shape and Pose from Images in the Wildcited by 190
- 2020ContactPose: A Dataset of Grasps with Object Contact and Hand Posecited by 183
- 2020Whole-Body Human Pose Estimation in the Wildcited by 178
- 20193D Hand Shape and Pose Estimation from a Single RGB Imagecited by 174
- 2021FrankMocap: A Monocular 3D Whole-Body Pose Estimation System via Regression and Integrationcited by 169
- 2021Hand-Object Contact Consistency Reasoning for Human Grasps Generationcited by 160
- 2017GANerated Hands for Real-time 3D Hand Tracking from Monocular RGBcited by 147
- 2020Weakly-Supervised Mesh-Convolutional Hand Reconstruction in the Wildcited by 146
- 2020Reconstructing Hand-Object Interactions in the Wildcited by 144
- 2019H+O: Unified Egocentric Recognition of 3D Hand-Object Poses and Interactionscited by 141
- 2022EPIC-KITCHENS VISOR Benchmark: VIdeo Segmentations and Object Relationscited by 140
- 2020Leveraging Photometric Consistency over Time for Sparsely Supervised Hand-Object Reconstructioncited by 139
- 2021ContactOpt: Optimizing Contact to Improve Graspscited by 139
- 2022OakInk: A Large-scale Knowledge Repository for Understanding Hand-Object Interactioncited by 131
- 2024HOT3D: Hand and Object Tracking in 3D from Egocentric Multi-View Videoscited by 129
- 2020CPF: Learning a Contact Potential Field to Model the Hand-Object Interactioncited by 128
- 2019End-to-end Hand Mesh Recovery from a Monocular RGB Imagecited by 124
- 2024WiLoR: End-to-end 3D Hand Localization and Reconstruction in-the-wildcited by 124
- 2018Hand Pose Estimation via Latent 2.5D Heatmap Regressioncited by 111
- 2022What's in your hands? 3D Reconstruction of Generic Objects in Handscited by 90
- 2022Interacting Attention Graph for Single Image Two-Hand Reconstructioncited by 87
- 2023HOLD: Category-agnostic 3D Reconstruction of Interacting Hands and Objects from Videocited by 78
- 2021ArtiBoost: Boosting Articulated 3D Hand-Object Pose Estimation via Online Exploration and Synthesiscited by 76
- 2023Diffusion-Guided Reconstruction of Everyday Hand-Object Interaction Clipscited by 74
- 2025HaWoR: World-Space Hand Motion Reconstruction from Egocentric Videoscited by 66
- 2021Towards unconstrained joint hand-object reconstruction from RGB videoscited by 58
- 2022Collaborative Learning for Hand and Object Reconstruction with Attention-guided Graph Convolutioncited by 54
- 2023DECO: Dense Estimation of 3D Human-Scene Contact In The Wildcited by 54
- 2023LEMON: Learning 3D Human-Object Interaction Relation from 2D Imagescited by 49
50 most cited of 69 All 69 (JSON)
Built on by
Citing papers per year: 2026: 2.
2 papers.