# Physical Interaction: Reconstructing Hand-object Interactions with Physics

arXiv: https://arxiv.org/abs/2209.10833 · HTML: https://mlsanity.com/paper/2209.10833/

## Key facts

- Cited by 19 papers in the map (data through 14 Oct 2026).
- Builds on 21 of its 43 references that are in the map.
- First posted on arXiv 22 Sep 2022.

## Builds on (most cited first, 21 of 21)

- [Learning joint reconstruction of hands and manipulated objects](https://mlsanity.com/paper/1904.05767/) (2019)
- [DeepMimic: Example-Guided Deep Reinforcement Learning of Physics-Based Character Skills](https://mlsanity.com/paper/1804.02717/) (2018)
- [Physical Inertial Poser (PIP): Physics-aware Real-time Human Motion Tracking from Sparse Inertial Sensors](https://mlsanity.com/paper/2203.08528/) (2022)
- [H+O: Unified Egocentric Recognition of 3D Hand-Object Poses and Interactions](https://mlsanity.com/paper/1904.05349/) (2019)
- [ContactOpt: Optimizing Contact to Improve Grasps](https://mlsanity.com/paper/2104.07267/) (2021)
- [Capturing Hands in Action using Discriminative Salient Points and Physics Simulation](https://mlsanity.com/paper/1506.02178/) (2015)
- [SimPoE: Simulated Character Control for 3D Human Pose Estimation](https://mlsanity.com/paper/2104.00683/) (2021)
- [D-Grasp: Physically Plausible Dynamic Grasp Synthesis for Hand-Object Interactions](https://mlsanity.com/paper/2112.03028/) (2021)
- [Contact and Human Dynamics from Monocular Video](https://mlsanity.com/paper/2007.11678/) (2020)
- [Ego-Pose Estimation and Forecasting as Real-Time PD Control](https://mlsanity.com/paper/1906.03173/) (2019)
- [Neural Monocular 3D Human Motion Capture with Physical Awareness](https://mlsanity.com/paper/2105.01057/) (2021)
- [Real-time Joint Tracking of a Hand Manipulating an Object from RGB-D Input](https://mlsanity.com/paper/1610.04889/) (2016)
- [PhysCap: Physically Plausible Monocular 3D Motion Capture in Real Time](https://mlsanity.com/paper/2008.08880/) (2020)
- [Estimating 3D Motion and Forces of Person-Object Interactions from Monocular Video](https://mlsanity.com/paper/1904.02683/) (2019)
- [Use the Force, Luke! Learning to Predict Physical Forces by Simulating Effects](https://mlsanity.com/paper/2003.12045/) (2020)
- [Optical Non-Line-of-Sight Physics-based 3D Human Pose Estimation](https://mlsanity.com/paper/2003.14414/) (2020)
- [Learning to Use Chopsticks in Diverse Gripping Styles](https://mlsanity.com/paper/2205.14313/) (2022)
- [SFV: Reinforcement Learning of Physical Skills from Videos](https://mlsanity.com/paper/1810.03599/) (2018)
- [Weakly-supervised Learning of Human Dynamics](https://mlsanity.com/paper/2007.08969/) (2020)
- [Back to RGB: 3D tracking of hands and hand-object interactions based on short-baseline stereo](https://mlsanity.com/paper/1705.05301/) (2017)
- [Physically Plausible Pose Refinement using Fully Differentiable Forces](https://mlsanity.com/paper/2105.08196/) (2021)

## Built on by (most cited first, 19 of 19)

- [Recent Trends in 3D Reconstruction of General Non-Rigid Scenes](https://mlsanity.com/paper/2403.15064/) (2024)
- [Decaf: Monocular Deformation Capture for Face and Hand Interactions](https://mlsanity.com/paper/2309.16670/) (2023)
- [Dense Hand-Object(HO) GraspNet with Full Grasping Taxonomy and Dynamics](https://mlsanity.com/paper/2409.04033/) (2024)
- [MACS: Mass Conditioned 3D Hand and Object Motion Synthesis](https://mlsanity.com/paper/2312.14929/) (2023)
- [Learning Explicit Contact for Implicit Reconstruction of Hand-held Objects from Monocular Images](https://mlsanity.com/paper/2305.20089/) (2023)
- [Dynamic Hyperbolic Attention Network for Fine Hand-object Reconstruction](https://mlsanity.com/paper/2309.02965/) (2023)
- [Few-Shot Physically-Aware Articulated Mesh Generation via Hierarchical Deformation](https://mlsanity.com/paper/2308.10898/) (2023)
- [DeepSimHO: Stable Pose Estimation for Hand-Object Interaction via Physics Simulation](https://mlsanity.com/paper/2310.07206/) (2023)
- [Physics-aware Hand-object Interaction Denoising](https://mlsanity.com/paper/2405.11481/) (2024)
- [Ins-HOI: Instance Aware Human-Object Interactions Recovery](https://mlsanity.com/paper/2312.09641/) (2023)
- [Hand-Object Interaction Controller (HOIC): Deep Reinforcement Learning for Reconstructing Interactions with Physics](https://mlsanity.com/paper/2405.02676/) (2024)
- [PAD-Hand: Physics-Aware Diffusion for Hand Motion Recovery](https://mlsanity.com/paper/2603.26068/) (2026)
- [HandyPriors: Physically Consistent Perception of Hand-Object Interactions with Differentiable Priors](https://mlsanity.com/paper/2311.16552/) (2023)
- [Grasp in Gaussians: Fast Monocular Reconstruction of Dynamic Hand-Object Interactions](https://mlsanity.com/paper/2604.12929/) (2026)
- [FunHOI: Annotation-Free 3D Hand-Object Interaction Generation via Functional Text Guidance](https://mlsanity.com/paper/2502.20805/) (2025)
- [Human Motion Prediction, Reconstruction, and Generation](https://mlsanity.com/paper/2502.15956/) (2025)
- [VPHO: Joint Visual-Physical Cue Learning and Aggregation for Hand-Object Pose Estimation](https://mlsanity.com/paper/2511.12030/) (2025)
- [DexterCap: An Affordable and Automated System for Capturing Dexterous Hand-Object Manipulation](https://mlsanity.com/paper/2601.05844/) (2026)
- [DynamicHOI: Coupled Dynamics for Physics-aware HOI Reconstruction](https://mlsanity.com/paper/2609.36454/) (2026)

## Abstract (arXiv)

Single view-based reconstruction of hand-object interaction is challenging due to the severe observation missing caused by occlusions. This paper proposes a physics-based method to better solve the ambiguities in the reconstruction. It first proposes a force-based dynamic model of the in-hand object, which not only recovers the unobserved contacts but also solves for plausible contact forces. Next, a confidence-based slide prevention scheme is proposed, which combines both the kinematic confidences and the contact forces to jointly model static and sliding contact motion. Qualitative and quantitative experiments show that the proposed technique reconstructs both physically plausible and more accurate hand-object interaction and estimates plausible contact forces in real-time with a single RGBD sensor.