mlsanityData to 7 Oct 2026

Estimating 3D Motion and Forces of Human-Object Interactions from Internet Videos

arXiv 2111.01591 ↗First posted 2 Nov 2021cs.CVcited by 9

Not on the map. Why?

Abstract (arXiv)

In this paper, we introduce a method to automatically reconstruct the 3D motion of a person interacting with an object from a single RGB video. Our method estimates the 3D poses of the person together with the object pose, the contact positions and the contact forces exerted on the human body. The main contributions of this work are three-fold. First, we introduce an approach to jointly estimate the motion and the actuation forces of the person on the manipulated object by modeling contacts and the dynamics of the interactions. This is cast as a large-scale trajectory optimization problem. Second, we develop a method to automatically recognize from the input video the 2D position and timing of contacts between the person and the object or the ground, thereby significantly simplifying the complexity of the optimization. Third, we validate our approach on a recent video+MoCap dataset capturing typical parkour actions, and demonstrate its performance on a new dataset of Internet videos showing people manipulating a variety of tools in unconstrained environments.

Key facts. Cited by 9 papers in the map (data through 7 Oct 2026). Builds on 31 of its 87 references that are in the map. First posted on arXiv 2 Nov 2021.

Builds on

31 of 87 references.

References · 31

  1. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  2. 2014Microsoft COCO: Common Objects in Contextcited by 17,498
  3. 2017Mask R-CNNcited by 7,259
  4. 2017Domain Randomization for Transferring Deep Neural Networks from Simulation to the Real Worldcited by 1,401
  5. 2016Realtime Multi-Person 2D Pose Estimation using Part Affinity Fieldscited by 1,061
  6. 2016Stacked Hourglass Networks for Human Pose Estimationcited by 913
  7. 2017End-to-end Recovery of Human Shape and Posecited by 883
  8. 2017PoseCNN: A Convolutional Neural Network for 6D Object Pose Estimation in Cluttered Scenescited by 768
  9. 2016Keep it SMPL: Automatic Estimation of 3D Human Pose and Shape from a Single Imagecited by 692
  10. 2019VIBE: Video Inference for Human Body Pose and Shape Estimationcited by 615
  11. 2018Learning 3D Human Dynamics from Videocited by 315
  12. 2018DeepIM: Deep Iterative Matching for 6D Pose Estimationcited by 283
19 more
  1. 2017BB8: A Scalable, Accurate, Robust to Partial Occlusion Method for Predicting the 3D Poses of Challenging Objects without Using Depthcited by 225
  2. 2016Coarse-to-Fine Volumetric Prediction for Single-Image 3D Human Posecited by 220
  3. 2016Associative Embedding: End-to-End Learning for Joint Detection and Groupingcited by 211
  4. 2018Monocular Total Capture: Posing Face, Body, and Hands in the Wildcited by 162
  5. 2015Unsupervised Learning from Narrated Instruction Videoscited by 116
  6. 20163D Human Pose Estimation = 2D Pose Estimation + Matchingcited by 112
  7. 2016DeeperCut: A Deeper, Stronger, and Faster Multi-Person Pose Estimation Modelcited by 96
  8. 2020Contact and Human Dynamics from Monocular Videocited by 94
  9. 2018Making Deep Heatmaps Robust to Partial Occlusions for 3D Object Pose Estimationcited by 74
  10. 20163D Human Pose Estimation from a Single Image via Distance Matrix Regressioncited by 73
  11. 2015Sparseness Meets Deepness: 3D Human Pose Estimation from Monocular Videocited by 54
  12. 2020PhysCap: Physically Plausible Monocular 3D Motion Capture in Real Timecited by 50
  13. 2019Estimating 3D Motion and Forces of Person-Object Interactions from Monocular Videocited by 48
  14. 2015What's Cookin'? Interpreting Cooking Videos using Text, Speech and Visioncited by 31
  15. 20183D Pose Estimation and 3D Model Retrieval for Objects in the Wildcited by 31
  16. 2017Going Further with Point Pair Featurescited by 28
  17. 2015Direct Prediction of 3D Body Poses from Motion Compensated Sequencescited by 26
  18. 2017Feature Mapping for Learning Fast and Accurate 3D Pose Inference from Synthetic Imagescited by 15
  19. 2019Virtual Training for a Real Application: Accurate Object-Robot Relative Localization without Calibrationcited by 5

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 1; 2023: 2; 2024: 1; 2025: 2; 2026: 3.

9 papers.

  1. 20256D Object Pose Tracking in Internet Videos for Robotic Manipulationcited by 10
  2. 2025Large-scale Pre-training for Grounded Video Caption Generationcited by 5
  3. 2022Imitrob: Imitation Learning Dataset for Training and Evaluating 6D Object Pose Estimatorscited by 3
  4. 2026Online Estimation and Manipulation of Articulated Objectscited by 3
  5. 2023Physics-Guided Human Motion Capture with Pose Probability Modelingcited by 2
  6. 2023Wild Motion Unleashed: Markerless 3D Kinematics and Force Estimation in Cheetahscited by 0
  7. 2024Grounded Video Caption Generationcited by 0
  8. 2026Reconstruction-Anchored Diffusion Model for Text-to-Motion Generationcited by 0
  9. 2026PACT: End-to-End Learning of Human Pose, Contacts, and Forces from Videocited by 0