mlsanityData to 7 Oct 2026

MoCapDeform: Monocular 3D Human Motion Capture in Deformable Scenes

arXiv 2208.08439 ↗First posted 17 Aug 2022cs.CVcited by 18

Not on the map. Why?

Abstract (arXiv)

3D human motion capture from monocular RGB images respecting interactions of a subject with complex and possibly deformable environments is a very challenging, ill-posed and under-explored problem. Existing methods address it only weakly and do not model possible surface deformations often occurring when humans interact with scene surfaces. In contrast, this paper proposes MoCapDeform, i.e., a new framework for monocular 3D human motion capture that is the first to explicitly model non-rigid deformations of a 3D scene for improved 3D human pose estimation and deformable environment reconstruction. MoCapDeform accepts a monocular RGB video and a 3D scene mesh aligned in the camera space. It first localises a subject in the input monocular video along with dense contact labels using a new raycasting based strategy. Next, our human-environment interaction constraints are leveraged to jointly optimise global 3D human poses and non-rigid surface deformations. MoCapDeform achieves superior accuracy than competing methods on several datasets, including our newly recorded one with deforming background scenes.

Key facts. Cited by 18 papers in the map (data through 7 Oct 2026). Builds on 49 of its 73 references that are in the map. First posted on arXiv 17 Aug 2022.

Builds on

49 of 73 references.

References · 49

  1. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  2. 2017ScanNet: Richly-annotated 3D Reconstructions of Indoor Scenescited by 3,088
  3. 2019Expressive Body Capture: 3D Hands, Face, and Body from a Single Imagecited by 1,600
  4. 2019AMASS: Archive of Motion Capture as Surface Shapescited by 1,208
  5. 2016Realtime Multi-Person 2D Pose Estimation using Part Affinity Fieldscited by 1,061
  6. 2017End-to-end Recovery of Human Shape and Posecited by 883
  7. 2018Open3D: A Modern Library for 3D Data Processingcited by 717
  8. 2016Keep it SMPL: Automatic Estimation of 3D Human Pose and Shape from a Single Imagecited by 692
  9. 2019VIBE: Video Inference for Human Body Pose and Shape Estimationcited by 615
  10. 2016Monocular 3D Human Pose Estimation In The Wild Using Improved CNN Supervisioncited by 493
  11. 2019PointRend: Image Segmentation as Renderingcited by 439
  12. 2017A simple yet effective baseline for 3d human pose estimationcited by 401
37 more
  1. 2016Convolutional Pose Machinescited by 325
  2. 2021HuMoR: 3D Human Motion Model for Robust Pose Estimationcited by 319
  3. 2019Resolving 3D Human Pose Ambiguities with 3D Scene Constraintscited by 294
  4. 2017Hand Keypoint Detection in Single Images using Multiview Bootstrappingcited by 231
  5. 2018Learning to Estimate 3D Human Pose and Shape from a Single Color Imagecited by 226
  6. 2016Coarse-to-Fine Volumetric Prediction for Single-Image 3D Human Posecited by 220
  7. 2020Long-term Human Motion Prediction with Scene Contextcited by 191
  8. 2020Populating 3D Scenes by Learning Human-Scene Interactioncited by 164
  9. 2020DeepCap: Monocular Human Performance Capture Using Weak Supervisioncited by 162
  10. 2020Perceiving 3D Human-Object Spatial Arrangements from a Single Image in the Wildcited by 152
  11. 2022Physical Inertial Poser (PIP): Physics-aware Real-time Human Motion Tracking from Sparse Inertial Sensorscited by 144
  12. 2021Human POSEitioning System (HPS): 3D Human Pose Estimation and Self-localization in Large Scenes from Body-Mounted Sensorscited by 137
  13. 2019Camera Distance-aware Top-down Approach for 3D Multi-person Pose Estimation from a Single RGB Imagecited by 131
  14. 2017VNect: Real-time 3D Human Pose Estimation with a Single RGB Cameracited by 129
  15. 2019Generating 3D People in Scenes without Peoplecited by 127
  16. 20163D Human Pose Estimation = 2D Pose Estimation + Matchingcited by 112
  17. 2021Learning Motion Priors for 4D Human Body Capture in 3D Scenescited by 98
  18. 2020Contact and Human Dynamics from Monocular Videocited by 94
  19. 2018LiveCap: Real-time Human Performance Capture from Monocular Videocited by 93
  20. 20163D Human Pose Estimation from a Single Image via Distance Matrix Regressioncited by 73
  21. 2017Lifting from the Deep: Convolutional 3D Pose Estimation from a Single Imagecited by 73
  22. 2018Unsupervised Geometry-Aware Representation for 3D Human Pose Estimationcited by 73
  23. 2021Neural Monocular 3D Human Motion Capture with Physical Awarenesscited by 70
  24. 2019XNect: Real-time Multi-Person 3D Motion Capture with a Single RGB Cameracited by 68
  25. 2019EventCap: Monocular 3D Capture of High-Speed Human Motions using an Event Cameracited by 65
  26. 2022UnrealEgo: A New Dataset for Robust Egocentric 3D Human Motion Capturecited by 56
  27. 2020Holistic 3D Human and Scene Mesh Estimation from Single View Imagescited by 54
  28. 2020Monocular Real-time Full Body Capture with Inter-part Correlationscited by 53
  29. 2020PhysCap: Physically Plausible Monocular 3D Motion Capture in Real Timecited by 50
  30. 2019In the Wild Human Pose Estimation Using Explicit 2D Features and Intermediate 3D Representationscited by 48
  31. 2016Structured Prediction of 3D Human Pose with Deep Neural Networkscited by 43
  32. 2016Deep Kinematic Pose Regressioncited by 40
  33. 2021ElePose: Unsupervised 3D Human Pose Estimation by Predicting Camera Elevation and Learning Normalizing Flows on 2D Posescited by 35
  34. 2020MotioNet: 3D Human Motion Reconstruction from Monocular Video with Skeleton Consistencycited by 31
  35. 2021Gravity-Aware Monocular 3D Human-Object Reconstructioncited by 30
  36. 2019Multi-Person 3D Human Pose Estimation from Monocular Imagescited by 19
  37. 2018iMapper: Interaction-guided Joint Scene and Human Motion Mapping from Monocular Videoscited by 6

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 2; 2023: 7; 2024: 4; 2025: 3; 2026: 2.

18 papers.

Sort:

By area:

  1. 2023Visibility Aware Human-Object Interaction Tracking from Single RGB Cameracited by 58
  2. 2022TORE: Token Reduction for Efficient Human Mesh Recovery with Transformercited by 41
  3. 2023Detecting Human-Object Contact in Imagescited by 41
  4. 2024CORE4D: A 4D Human-Object-Human Interaction Dataset for Collaborative Object REarrangementcited by 40
  5. 2023Template Free Reconstruction of Human-object Interaction with Procedural Interaction Generationcited by 37
  6. 2023Object pop-up: Can we infer 3D objects and their poses from human interactions alone?cited by 36
  7. 2022Scene-aware Egocentric 3D Human Pose Estimationcited by 30
  8. 2024Recent Trends in 3D Reconstruction of General Non-Rigid Scenescited by 28
  9. 2023Decaf: Monocular Deformation Capture for Face and Hand Interactionscited by 17
  10. 2023Learning Human Mesh Recovery in 3D Scenescited by 13
  11. 2024DICE: End-to-end Deformation Capture of Hand-Face Interactions from a Single Imagecited by 9
  12. 2025ODHSR: Online Dense 3D Reconstruction of Humans and Scenes from Monocular Videoscited by 8
6 more
  1. 2024Text to Blind Motioncited by 3
  2. 2023Reconstructing 3D Human Pose from RGB-D Data with Occlusionscited by 2
  3. 2025Betsu-Betsu: Multi-View Separable 3D Reconstruction of Two Interacting Objectscited by 2
  4. 2025Generative Modeling of Shape-Dependent Self-Contact Human Posescited by 2
  5. 2026GraphiContact: Pose-aware Human-Scene Robust Contact Perception for Interactive Systemscited by 1
  6. 2026RHINO: Reconstructing Human Interactions with Novel Objects from Monocular Videoscited by 0