# PhysCap: Physically Plausible Monocular 3D Motion Capture in Real Time

arXiv: https://arxiv.org/abs/2008.08880 · HTML: https://mlsanity.com/paper/2008.08880/

## Key facts

- Cited by 50 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 20 Aug 2020.

## Built on by (most cited first, 30 of 50)

- [MoFusion: A Framework for Denoising-Diffusion-based Motion Synthesis](https://mlsanity.com/paper/2212.04495/) (2022)
- [Synthesis of Compositional Animations from Textual Descriptions](https://mlsanity.com/paper/2103.14675/) (2021)
- [Decoupling Human and Camera Motion from Videos in the Wild](https://mlsanity.com/paper/2302.12827/) (2023)
- [D-Grasp: Physically Plausible Dynamic Grasp Synthesis for Hand-Object Interactions](https://mlsanity.com/paper/2112.03028/) (2021)
- [3D Human Pose Estimation via Intuitive Physics](https://mlsanity.com/paper/2303.18246/) (2023)
- [Dynamics-Regulated Kinematic Policy for Egocentric Pose Estimation](https://mlsanity.com/paper/2106.05969/) (2021)
- [Probabilistic Monocular 3D Human Pose Estimation with Normalizing Flows](https://mlsanity.com/paper/2107.13788/) (2021)
- [Neural Monocular 3D Human Motion Capture with Physical Awareness](https://mlsanity.com/paper/2105.01057/) (2021)
- [MoConVQ: Unified Physics-Based Motion Control via Scalable Discrete Representations](https://mlsanity.com/paper/2310.10198/) (2023)
- [ArtiGrasp: Physically Plausible Synthesis of Bi-Manual Dexterous Grasping and Articulation](https://mlsanity.com/paper/2309.03891/) (2023)
- [Differentiable Dynamics for Articulated 3d Human Motion Reconstruction](https://mlsanity.com/paper/2205.12256/) (2022)
- [Trajectory Optimization for Physics-Based Reconstruction of 3d Human Pose from Monocular Video](https://mlsanity.com/paper/2205.12292/) (2022)
- [PACE: Human and Camera Motion Estimation from in-the-wild Videos](https://mlsanity.com/paper/2310.13768/) (2023)
- [Neural MoCon: Neural Motion Control for Physically Plausible Human Motion Capture](https://mlsanity.com/paper/2203.14065/) (2022)
- [Gravity-Aware Monocular 3D Human-Object Reconstruction](https://mlsanity.com/paper/2108.08844/) (2021)
- [State of the Art in Dense Monocular Non-Rigid 3D Reconstruction](https://mlsanity.com/paper/2210.15664/) (2022)
- [Scene-aware Egocentric 3D Human Pose Estimation](https://mlsanity.com/paper/2212.11684/) (2022)
- [HULC: 3D Human Motion Capture with Pose Manifold Sampling and Dense Contact Guidance](https://mlsanity.com/paper/2205.05677/) (2022)
- [Recent Trends in 3D Reconstruction of General Non-Rigid Scenes](https://mlsanity.com/paper/2403.15064/) (2024)
- [PhysPT: Physics-aware Pretrained Transformer for Estimating Human Dynamics from Monocular Videos](https://mlsanity.com/paper/2404.04430/) (2024)
- [Out-of-Domain Human Mesh Reconstruction via Dynamic Bilevel Online Adaptation](https://mlsanity.com/paper/2111.04017/) (2021)
- [PACER+: On-Demand Pedestrian Animation Controller in Driving Scenarios](https://mlsanity.com/paper/2404.19722/) (2024)
- [φ-SfT: Shape-from-Template with a Physics-Based Deformation Model](https://mlsanity.com/paper/2203.11938/) (2022)
- [Physical Interaction: Reconstructing Hand-object Interactions with Physics](https://mlsanity.com/paper/2209.10833/) (2022)
- [Scene-Aware 3D Multi-Human Motion Capture from a Single Camera](https://mlsanity.com/paper/2301.05175/) (2023)
- [MoCapDeform: Monocular 3D Human Motion Capture in Deformable Scenes](https://mlsanity.com/paper/2208.08439/) (2022)
- [Fusing Monocular Images and Sparse IMU Signals for Real-time Human Motion Capture](https://mlsanity.com/paper/2309.00310/) (2023)
- [Transformer-Based Learned Optimization](https://mlsanity.com/paper/2212.01055/) (2022)
- [Learning Human Mesh Recovery in 3D Scenes](https://mlsanity.com/paper/2306.03847/) (2023)
- [DivaTrack: Diverse Bodies and Motions from Acceleration-Enhanced Three-Point Trackers](https://mlsanity.com/paper/2402.09211/) (2024)

## Abstract (arXiv)

Marker-less 3D human motion capture from a single colour camera has seen significant progress. However, it is a very challenging and severely ill-posed problem. In consequence, even the most accurate state-of-the-art approaches have significant limitations. Purely kinematic formulations on the basis of individual joints or skeletons, and the frequent frame-wise reconstruction in state-of-the-art methods greatly limit 3D accuracy and temporal stability compared to multi-view or marker-based motion capture. Further, captured 3D poses are often physically incorrect and biomechanically implausible, or exhibit implausible environment interactions (floor penetration, foot skating, unnatural body leaning and strong shifting in depth), which is problematic for any use case in computer graphics. We, therefore, present PhysCap, the first algorithm for physically plausible, real-time and marker-less human 3D motion capture with a single colour camera at 25 fps. Our algorithm first captures 3D human poses purely kinematically. To this end, a CNN infers 2D and 3D joint positions, and subsequently, an inverse kinematics step finds space-time coherent joint angles and global 3D pose. Next, these kinematic reconstructions are used as constraints in a real-time physics-based pose optimiser that accounts for environment constraints (e.g., collision handling and floor placement), gravity, and biophysical plausibility of human postures. Our approach employs a combination of ground reaction force and residual force for plausible root control, and uses a trained neural network to detect foot contact events in images. Our method captures physically plausible and temporally stable global 3D human motion, without physically implausible postures, floor penetrations or foot skating, from video in real time and in general scenes. The video is available at http://gvv.mpi-inf.mpg.de/projects/PhysCap