# EgoPHI: Estimating 3D Hand-Object Contact and Force from Egocentric Vision

arXiv: https://arxiv.org/abs/2608.13014 · HTML: https://mlsanity.com/paper/2608.13014/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 35 of its 51 references that are in the map.
- First posted on arXiv 13 Aug 2026.

## Builds on (most cited first, 30 of 35)

- [Expressive Body Capture: 3D Hands, Face, and Body from a Single Image](https://mlsanity.com/paper/1904.05866/) (2019)
- [GRAB: A Dataset of Whole-Body Human Grasping of Objects](https://mlsanity.com/paper/2008.11200/) (2020)
- [Learning joint reconstruction of hands and manipulated objects](https://mlsanity.com/paper/1904.05767/) (2019)
- [HOI4D: A 4D Egocentric Dataset for Category-Level Human-Object Interaction](https://mlsanity.com/paper/2203.01577/) (2022)
- [DexYCB: A Benchmark for Capturing Hand Grasping of Objects](https://mlsanity.com/paper/2104.04631/) (2021)
- [ARCTIC: A Dataset for Dexterous Bimanual Hand-Object Manipulation](https://mlsanity.com/paper/2204.13662/) (2022)
- [Reconstructing Hands in 3D with Transformers](https://mlsanity.com/paper/2312.05251/) (2023)
- [H2O: Two Hands Manipulating Objects for First Person Interaction Recognition](https://mlsanity.com/paper/2104.11181/) (2021)
- [SAM 3D: 3Dfy Anything in Images](https://mlsanity.com/paper/2511.16624/) (2025)
- [ContactPose: A Dataset of Grasps with Object Contact and Hand Pose](https://mlsanity.com/paper/2007.09545/) (2020)
- [Capturing and Inferring Dense Full-Body Human-Scene Contact](https://mlsanity.com/paper/2206.09553/) (2022)
- [Reconstructing Hand-Object Interactions in the Wild](https://mlsanity.com/paper/2012.09856/) (2020)
- [Visibility Aware Human-Object Interaction Tracking from Single RGB Camera](https://mlsanity.com/paper/2303.16479/) (2023)
- [DECO: Dense Estimation of 3D Human-Scene Contact In The Wild](https://mlsanity.com/paper/2309.15273/) (2023)
- [Estimating 3D Motion and Forces of Person-Object Interactions from Monocular Video](https://mlsanity.com/paper/1904.02683/) (2019)
- [Detecting Human-Object Contact in Images](https://mlsanity.com/paper/2303.03373/) (2023)
- [Use the Force, Luke! Learning to Predict Physical Forces by Simulating Effects](https://mlsanity.com/paper/2003.12045/) (2020)
- [Beyond the Contact: Discovering Comprehensive Affordance for 3D Objects from Pre-trained 2D Diffusion Models](https://mlsanity.com/paper/2401.12978/) (2024)
- [HO-3D_v3: Improving the Accuracy of Hand-Object Annotations of the HO-3D Dataset](https://mlsanity.com/paper/2107.00887/) (2021)
- [PressureVision: Estimating Hand Pressure from a Single RGB Image](https://mlsanity.com/paper/2203.10385/) (2022)
- [PICO: Reconstructing 3D People In Contact with Objects](https://mlsanity.com/paper/2504.17695/) (2025)
- [EgoPressure: A Dataset for Hand Pressure and Pose Estimation in Egocentric Vision](https://mlsanity.com/paper/2409.02224/) (2024)
- [Learning Dense Hand Contact Estimation from Imbalanced Data](https://mlsanity.com/paper/2505.11152/) (2025)
- [PressureVision++: Estimating Fingertip Pressure from Diverse RGB Images](https://mlsanity.com/paper/2301.02310/) (2023)
- [HMDO: Markerless Multi-view Hand Manipulation Capture with Deformable Objects](https://mlsanity.com/paper/2301.07652/) (2023)
- [Learning Explicit Contact for Implicit Reconstruction of Hand-held Objects from Monocular Images](https://mlsanity.com/paper/2305.20089/) (2023)
- [Dynamic Reconstruction of Hand-Object Interaction with Distributed Force-aware Contact Representation](https://mlsanity.com/paper/2411.09572/) (2024)
- [DyTact: Capturing Dynamic Contacts in Hand-Object Manipulation](https://mlsanity.com/paper/2506.03103/) (2025)
- [Contact-Implicit Model Predictive Control for Dexterous In-hand Manipulation: A Long-Horizon and Robust Approach](https://mlsanity.com/paper/2402.18897/) (2024)
- [FORCE: Physics-aware Human-object Interaction](https://mlsanity.com/paper/2403.11237/) (2024)

## Abstract (arXiv)

Understanding hand-object interaction from egocentric vision is essential for modeling how people physically engage with the surrounding world. Yet reasoning about physically grounded interaction requires estimating the forces acting on hands and objects, beyond localizing contact. We present EgoPHI, the first method that jointly estimates dense contact maps and 3D force distributions on hand and object meshes from a single monocular RGB image and object geometry. To address the lack of scalable ground-truth force annotations, we introduce a physics-based simulation pipeline that augments existing hand-object datasets with dense per-vertex force supervision. EgoPHI then learns dense 3D contact and force on interacting hand and articulated object meshes, extending vision-based force estimation beyond image-space or planar settings. Our evaluation on in-distribution and out-of-distribution benchmarks shows that EgoPHI improves force estimation over existing approaches while generalizing to unseen datasets. To evaluate sim-to-real transfer, we constructed two physical objects that capture dense object contact and force magnitude and used them to record a dataset of interactions from eight participants across diverse touch and grasp types. Our results demonstrate that EgoPHI recovers meaningful 3D contact and force distributions in simulated, out-of-distribution, and real-world settings, advancing egocentric hand-object understanding from contact localization toward physically grounded interaction reasoning.