mlsanityData to 14 Oct 2026

Estimating 3D Motion and Forces of Person-Object Interactions from Monocular Video

arXiv 1904.02683 ↗First posted 4 Apr 2019cs.CVcited by 48

Not on the map. Why?

Abstract (arXiv)

In this paper, we introduce a method to automatically reconstruct the 3D motion of a person interacting with an object from a single RGB video. Our method estimates the 3D poses of the person and the object, contact positions, and forces and torques actuated by the human limbs. The main contributions of this work are three-fold. First, we introduce an approach to jointly estimate the motion and the actuation forces of the person on the manipulated object by modeling contacts and the dynamics of their interactions. This is cast as a large-scale trajectory optimization problem. Second, we develop a method to automatically recognize from the input video the position and timing of contacts between the person and the object or the ground, thereby significantly simplifying the complexity of the optimization. Third, we validate our approach on a recent MoCap dataset with ground truth contact forces and demonstrate its performance on a new dataset of Internet videos showing people manipulating a variety of tools in unconstrained environments.

Key facts. Cited by 48 papers in the map (data through 14 Oct 2026). First posted on arXiv 4 Apr 2019.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2021: 9; 2022: 14; 2023: 4; 2024: 9; 2025: 6; 2026: 6.

48 papers.

Sort:

By area:

  1. 2021HuMoR: 3D Human Motion Model for Robust Pose Estimationcited by 324
  2. 2021Human POSEitioning System (HPS): 3D Human Pose Estimation and Self-localization in Large Scenes from Body-Mounted Sensorscited by 137
  3. 2021SimPoE: Simulated Character Control for 3D Human Pose Estimationcited by 114
  4. 2021Learning Motion Priors for 4D Human Body Capture in 3D Scenescited by 98
  5. 2022CHORE: Contact, Human and Object REconstruction from a single RGB imagecited by 95
  6. 2021On Self-Contact and Human Posecited by 89
  7. 2021Neural Monocular 3D Human Motion Capture with Physical Awarenesscited by 71
  8. 2021Physics-based Human Motion Estimation and Synthesis from Videoscited by 66
  9. 2024Estimating Body and Hand Motion in an Ego-sensed Worldcited by 58
  10. 2022Human-Aware Object Placement for Visual Environment Reconstructioncited by 56
  11. 2023Detecting Human-Object Contact in Imagescited by 41
  12. 2022Trajectory Optimization for Physics-Based Reconstruction of 3d Human Pose from Monocular Videocited by 37
36 more
  1. 2024Beyond the Contact: Discovering Comprehensive Affordance for 3D Objects from Pre-trained 2D Diffusion Modelscited by 32
  2. 2022Neural MoCon: Neural Motion Control for Physically Plausible Human Motion Capturecited by 31
  3. 2021Gravity-Aware Monocular 3D Human-Object Reconstructioncited by 30
  4. 2022HULC: 3D Human Motion Capture with Pose Manifold Sampling and Dense Contact Guidancecited by 29
  5. 2024PhysPT: Physics-aware Pretrained Transformer for Estimating Human Dynamics from Monocular Videoscited by 28
  6. 2022PressureVision: Estimating Hand Pressure from a Single RGB Imagecited by 23
  7. 2022Learn to Predict How Humans Manipulate Large-sized Objects from Interactive Motionscited by 21
  8. 2024Programmable Motion Generation for Open-Set Motion Control Taskscited by 20
  9. 2022Physical Interaction: Reconstructing Hand-object Interactions with Physicscited by 19
  10. 2023CHORUS: Learning Canonicalized 3D Human-Object Spatial Relations from Unbounded Synthesized Imagescited by 19
  11. 2024EgoPressure: A Dataset for Hand Pressure and Pose Estimation in Egocentric Visioncited by 19
  12. 2022Learning Variational Motion Prior for Video-based Motion Capturecited by 17
  13. 2022Transformer-Based Learned Optimizationcited by 13
  14. 2023PressureVision++: Estimating Fingertip Pressure from Diverse RGB Imagescited by 12
  15. 2024AddBiomechanics Dataset: Capturing the Physics of Human Motion at Scalecited by 11
  16. 2022Knowledge-augmented Deep Learning and Its Applications: A Surveycited by 10
  17. 2021Estimating 3D Motion and Forces of Human-Object Interactions from Internet Videoscited by 9
  18. 2023Compositional 3D Human-Object Neural Animationcited by 9
  19. 2025ScoreHOI: Physically Plausible Reconstruction of Human-Object Interaction via Score-Guided Diffusioncited by 9
  20. 2025Zero-Shot Human-Object Interaction Synthesis with Multimodal Priorscited by 7
  21. 2025Seeing the Wind from a Falling Leafcited by 7
  22. 2024TouchInsight: Uncertainty-aware Rapid Touch and Text Input for Mixed Reality from Egocentric Visioncited by 4
  23. 2026FEEL (Force-Enhanced Egocentric Learning): A Dataset for Physical Action Understandingcited by 4
  24. 2026PAD-Hand: Physics-Aware Diffusion for Hand Motion Recoverycited by 4
  25. 2022Learning to Estimate External Forces of Human Motion in Videocited by 3
  26. 2024Posture-Informed Muscular Force Learning for Robust Hand Pressure Estimationcited by 3
  27. 2022Image-based Stability Quantificationcited by 2
  28. 2026PhysMoDPO: Physically-Plausible Humanoid Motion with Preference Optimizationcited by 2
  29. 2025Shape-aware Inertial Poser: Motion Tracking for Humans with Diverse Shapes Using Sparse Inertial Sensorscited by 1
  30. 2025FootFormer: Estimating Stability from Visual Inputcited by 1
  31. 2025Beyond Motion Pattern: An Empirical Study of Physical Forces for Human Motion Understandingcited by 1
  32. 2026Grasp-and-Lift: Executable 3D Hand-Object Interaction Reconstruction via Physics-in-the-Loop Optimizationcited by 1
  33. 2022Unified Simulation, Perception, and Generation of Human Behaviorcited by 0
  34. 2024PoseAugment: Generative Human Pose Data Augmentation with Physical Plausibility for IMU-based Motion Capturecited by 0
  35. 2026EgoPHI: Estimating 3D Hand-Object Contact and Force from Egocentric Visioncited by 0
  36. 2026PACT: End-to-End Learning of Human Pose, Contacts, and Forces from Videocited by 0