# Neural MoCon: Neural Motion Control for Physically Plausible Human Motion Capture

arXiv: https://arxiv.org/abs/2203.14065 · HTML: https://mlsanity.com/paper/2203.14065/

## Key facts

- Cited by 31 papers in the map (data through 14 Oct 2026).
- Builds on 30 of its 72 references that are in the map.
- First posted on arXiv 26 Mar 2022.

## Builds on (most cited first, 30 of 30)

- [PyTorch: An Imperative Style, High-Performance Deep Learning Library](https://mlsanity.com/paper/1912.01703/) (2019)
- [Expressive Body Capture: 3D Hands, Face, and Body from a Single Image](https://mlsanity.com/paper/1904.05866/) (2019)
- [End-to-end Recovery of Human Shape and Pose](https://mlsanity.com/paper/1712.06584/) (2017)
- [Learning to Reconstruct 3D Human Pose and Shape via Model-fitting in the Loop](https://mlsanity.com/paper/1909.12828/) (2019)
- [VIBE: Video Inference for Human Body Pose and Shape Estimation](https://mlsanity.com/paper/1912.05656/) (2019)
- [The CMA Evolution Strategy: A Tutorial](https://mlsanity.com/paper/1604.00772/) (2016)
- [RMPE: Regional Multi-person Pose Estimation](https://mlsanity.com/paper/1612.00137/) (2016)
- [Deep Reinforcement Learning: An Overview](https://mlsanity.com/paper/1701.07274/) (2017)
- [HuMoR: 3D Human Motion Model for Robust Pose Estimation](https://mlsanity.com/paper/2105.04668/) (2021)
- [Resolving 3D Human Pose Ambiguities with 3D Scene Constraints](https://mlsanity.com/paper/1908.06963/) (2019)
- [Challenges of Real-World Reinforcement Learning](https://mlsanity.com/paper/1904.12901/) (2019)
- [Stochastic Scene-Aware Motion Prediction](https://mlsanity.com/paper/2108.08284/) (2021)
- [Long-term Human Motion Prediction with Scene Context](https://mlsanity.com/paper/2007.03672/) (2020)
- [DeepHuman: 3D Human Reconstruction from a Single Image](https://mlsanity.com/paper/1903.06473/) (2019)
- [Populating 3D Scenes by Learning Human-Scene Interaction](https://mlsanity.com/paper/2012.11581/) (2020)
- [Coherent Reconstruction of Multiple Humans from a Single Image](https://mlsanity.com/paper/2006.08586/) (2020)
- [SimPoE: Simulated Character Control for 3D Human Pose Estimation](https://mlsanity.com/paper/2104.00683/) (2021)
- [Learning Motion Priors for 4D Human Body Capture in 3D Scenes](https://mlsanity.com/paper/2108.10399/) (2021)
- [Contact and Human Dynamics from Monocular Video](https://mlsanity.com/paper/2007.11678/) (2020)
- [Ego-Pose Estimation and Forecasting as Real-Time PD Control](https://mlsanity.com/paper/1906.03173/) (2019)
- [Neural Monocular 3D Human Motion Capture with Physical Awareness](https://mlsanity.com/paper/2105.01057/) (2021)
- [Physics-based Human Motion Estimation and Synthesis from Videos](https://mlsanity.com/paper/2109.09913/) (2021)
- [Residual Force Control for Agile Human Behavior Imitation and Extended Motion Synthesis](https://mlsanity.com/paper/2006.07364/) (2020)
- [ALLSTEPS: Curriculum-driven Learning of Stepping Stone Skills](https://mlsanity.com/paper/2005.04323/) (2020)
- [PhysCap: Physically Plausible Monocular 3D Motion Capture in Real Time](https://mlsanity.com/paper/2008.08880/) (2020)
- [Estimating 3D Motion and Forces of Person-Object Interactions from Monocular Video](https://mlsanity.com/paper/1904.02683/) (2019)
- [Dynamic Multi-Person Mesh Recovery From Uncalibrated Multi-View Cameras](https://mlsanity.com/paper/2110.10355/) (2021)
- [SFV: Reinforcement Learning of Physical Skills from Videos](https://mlsanity.com/paper/1810.03599/) (2018)
- [Revitalizing Optimization for 3D Human Pose and Shape Estimation: A Sparse Constrained Formulation](https://mlsanity.com/paper/2105.13965/) (2021)
- [Weakly-supervised Learning of Human Dynamics](https://mlsanity.com/paper/2007.08969/) (2020)

## Built on by (most cited first, 30 of 31)

- [Perpetual Humanoid Control for Real-time Simulated Avatars](https://mlsanity.com/paper/2305.06456/) (2023)
- [Physics-Informed Machine Learning: A Survey on Problems, Methods and Applications](https://mlsanity.com/paper/2211.08064/) (2022)
- [Synthesizing Physically Plausible Human Motions in 3D Scenes](https://mlsanity.com/paper/2308.09036/) (2023)
- [RoHM: Robust Human Motion Reconstruction via Diffusion](https://mlsanity.com/paper/2401.08570/) (2024)
- [Embodied Scene-aware Human Pose Estimation](https://mlsanity.com/paper/2206.09106/) (2022)
- [Physics-Informed Computer Vision: A Review and Perspectives](https://mlsanity.com/paper/2305.18035/) (2023)
- [PhysPT: Physics-aware Pretrained Transformer for Estimating Human Dynamics from Monocular Videos](https://mlsanity.com/paper/2404.04430/) (2024)
- [HMD-Poser: On-Device Real-time Human Motion Tracking from Scalable Sparse Observations](https://mlsanity.com/paper/2403.03561/) (2024)
- [Reconstructing Interacting Hands with Interaction Prior from Monocular Images](https://mlsanity.com/paper/2308.14082/) (2023)
- [Decaf: Monocular Deformation Capture for Face and Hand Interactions](https://mlsanity.com/paper/2309.16670/) (2023)
- [Real-Time Simulated Avatar from Head-Mounted Sensors](https://mlsanity.com/paper/2403.06862/) (2024)
- [Contact-aware Human Motion Generation from Textual Descriptions](https://mlsanity.com/paper/2403.15709/) (2024)
- [DICE: End-to-end Deformation Capture of Hand-Face Interactions from a Single Image](https://mlsanity.com/paper/2406.17988/) (2024)
- [ImDy: Human Inverse Dynamics from Imitated Observations](https://mlsanity.com/paper/2410.17610/) (2024)
- [FinePhys: Fine-grained Human Action Generation by Explicitly Incorporating Physical Laws for Effective Skeletal Guidance](https://mlsanity.com/paper/2505.13437/) (2025)
- [MOVIN: Real-time Motion Capture using a Single LiDAR](https://mlsanity.com/paper/2309.09314/) (2023)
- [Optimal-state Dynamics Estimation for Physics-based Human Motion Capture from Videos](https://mlsanity.com/paper/2410.07795/) (2024)
- [A Plug-and-Play Physical Motion Restoration Approach for In-the-Wild High-Difficulty Motions](https://mlsanity.com/paper/2412.17377/) (2024)
- [GraMMaR: Ground-aware Motion Model for 3D Human Motion Reconstruction](https://mlsanity.com/paper/2306.16736/) (2023)
- [Text to Blind Motion](https://mlsanity.com/paper/2412.05277/) (2024)
- [QuaMo: Quaternion Motions for Vision-based 3D Human Kinematics Capture](https://mlsanity.com/paper/2601.19580/) (2026)
- [Physics-Guided Human Motion Capture with Pose Probability Modeling](https://mlsanity.com/paper/2308.09910/) (2023)
- [Every Angle Is Worth A Second Glance: Mining Kinematic Skeletal Structures from Multi-view Joint Cloud](https://mlsanity.com/paper/2502.02936/) (2025)
- [Ground Reaction Inertial Poser: Physics-based Human Motion Capture from Sparse IMUs and Insole Pressure Sensors](https://mlsanity.com/paper/2603.16233/) (2026)
- [Motion-DVAE: Unsupervised learning for fast human motion denoising](https://mlsanity.com/paper/2306.05846/) (2023)
- [ELMO: Enhanced Real-time LiDAR Motion Capture through Upsampling](https://mlsanity.com/paper/2410.06963/) (2024)
- [Homogeneous Dynamics Space for Heterogeneous Humans](https://mlsanity.com/paper/2412.06146/) (2024)
- [Multi-person Physics-based Pose Estimation for Combat Sports](https://mlsanity.com/paper/2504.08175/) (2025)
- [Stability-Driven Motion Generation for Object-Guided Human-Human Co-Manipulation](https://mlsanity.com/paper/2604.20336/) (2026)
- [Gravity-guided Contact Dynamics Estimation from 3D Human Motions](https://mlsanity.com/paper/2606.08133/) (2026)

## Abstract (arXiv)

Due to the visual ambiguity, purely kinematic formulations on monocular human motion capture are often physically incorrect, biomechanically implausible, and can not reconstruct accurate interactions. In this work, we focus on exploiting the high-precision and non-differentiable physics simulator to incorporate dynamical constraints in motion capture. Our key-idea is to use real physical supervisions to train a target pose distribution prior for sampling-based motion control to capture physically plausible human motion. To obtain accurate reference motion with terrain interactions for the sampling, we first introduce an interaction constraint based on SDF (Signed Distance Field) to enforce appropriate ground contact modeling. We then design a novel two-branch decoder to avoid stochastic error from pseudo ground-truth and train a distribution prior with the non-differentiable physics simulator. Finally, we regress the sampling distribution from the current state of the physical character with the trained prior and sample satisfied target poses to track the estimated reference motion. Qualitative and quantitative results show that we can obtain physically plausible human motion with complex terrain interactions, human shape variations, and diverse behaviors. More information can be found at~urlhttps://www.yangangwang.com/papers/HBZ-NM-2022-03.html