# Estimating 3D Motion and Forces of Person-Object Interactions from Monocular Video

arXiv: https://arxiv.org/abs/1904.02683 · HTML: https://mlsanity.com/paper/1904.02683/

## Key facts

- Cited by 48 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 4 Apr 2019.

## Built on by (most cited first, 30 of 48)

- [HuMoR: 3D Human Motion Model for Robust Pose Estimation](https://mlsanity.com/paper/2105.04668/) (2021)
- [Human POSEitioning System (HPS): 3D Human Pose Estimation and Self-localization in Large Scenes from Body-Mounted Sensors](https://mlsanity.com/paper/2103.17265/) (2021)
- [SimPoE: Simulated Character Control for 3D Human Pose Estimation](https://mlsanity.com/paper/2104.00683/) (2021)
- [Learning Motion Priors for 4D Human Body Capture in 3D Scenes](https://mlsanity.com/paper/2108.10399/) (2021)
- [CHORE: Contact, Human and Object REconstruction from a single RGB image](https://mlsanity.com/paper/2204.02445/) (2022)
- [On Self-Contact and Human Pose](https://mlsanity.com/paper/2104.03176/) (2021)
- [Neural Monocular 3D Human Motion Capture with Physical Awareness](https://mlsanity.com/paper/2105.01057/) (2021)
- [Physics-based Human Motion Estimation and Synthesis from Videos](https://mlsanity.com/paper/2109.09913/) (2021)
- [Estimating Body and Hand Motion in an Ego-sensed World](https://mlsanity.com/paper/2410.03665/) (2024)
- [Human-Aware Object Placement for Visual Environment Reconstruction](https://mlsanity.com/paper/2203.03609/) (2022)
- [Detecting Human-Object Contact in Images](https://mlsanity.com/paper/2303.03373/) (2023)
- [Trajectory Optimization for Physics-Based Reconstruction of 3d Human Pose from Monocular Video](https://mlsanity.com/paper/2205.12292/) (2022)
- [Beyond the Contact: Discovering Comprehensive Affordance for 3D Objects from Pre-trained 2D Diffusion Models](https://mlsanity.com/paper/2401.12978/) (2024)
- [Neural MoCon: Neural Motion Control for Physically Plausible Human Motion Capture](https://mlsanity.com/paper/2203.14065/) (2022)
- [Gravity-Aware Monocular 3D Human-Object Reconstruction](https://mlsanity.com/paper/2108.08844/) (2021)
- [HULC: 3D Human Motion Capture with Pose Manifold Sampling and Dense Contact Guidance](https://mlsanity.com/paper/2205.05677/) (2022)
- [PhysPT: Physics-aware Pretrained Transformer for Estimating Human Dynamics from Monocular Videos](https://mlsanity.com/paper/2404.04430/) (2024)
- [PressureVision: Estimating Hand Pressure from a Single RGB Image](https://mlsanity.com/paper/2203.10385/) (2022)
- [Learn to Predict How Humans Manipulate Large-sized Objects from Interactive Motions](https://mlsanity.com/paper/2206.12612/) (2022)
- [Programmable Motion Generation for Open-Set Motion Control Tasks](https://mlsanity.com/paper/2405.19283/) (2024)
- [Physical Interaction: Reconstructing Hand-object Interactions with Physics](https://mlsanity.com/paper/2209.10833/) (2022)
- [CHORUS: Learning Canonicalized 3D Human-Object Spatial Relations from Unbounded Synthesized Images](https://mlsanity.com/paper/2308.12288/) (2023)
- [EgoPressure: A Dataset for Hand Pressure and Pose Estimation in Egocentric Vision](https://mlsanity.com/paper/2409.02224/) (2024)
- [Learning Variational Motion Prior for Video-based Motion Capture](https://mlsanity.com/paper/2210.15134/) (2022)
- [Transformer-Based Learned Optimization](https://mlsanity.com/paper/2212.01055/) (2022)
- [PressureVision++: Estimating Fingertip Pressure from Diverse RGB Images](https://mlsanity.com/paper/2301.02310/) (2023)
- [AddBiomechanics Dataset: Capturing the Physics of Human Motion at Scale](https://mlsanity.com/paper/2406.18537/) (2024)
- [Knowledge-augmented Deep Learning and Its Applications: A Survey](https://mlsanity.com/paper/2212.00017/) (2022)
- [Estimating 3D Motion and Forces of Human-Object Interactions from Internet Videos](https://mlsanity.com/paper/2111.01591/) (2021)
- [Compositional 3D Human-Object Neural Animation](https://mlsanity.com/paper/2304.14070/) (2023)

## Abstract (arXiv)

In this paper, we introduce a method to automatically reconstruct the 3D motion of a person interacting with an object from a single RGB video. Our method estimates the 3D poses of the person and the object, contact positions, and forces and torques actuated by the human limbs. The main contributions of this work are three-fold. First, we introduce an approach to jointly estimate the motion and the actuation forces of the person on the manipulated object by modeling contacts and the dynamics of their interactions. This is cast as a large-scale trajectory optimization problem. Second, we develop a method to automatically recognize from the input video the position and timing of contacts between the person and the object or the ground, thereby significantly simplifying the complexity of the optimization. Third, we validate our approach on a recent MoCap dataset with ground truth contact forces and demonstrate its performance on a new dataset of Internet videos showing people manipulating a variety of tools in unconstrained environments.