# Differentiable Dynamics for Articulated 3d Human Motion Reconstruction

arXiv: https://arxiv.org/abs/2205.12256 · HTML: https://mlsanity.com/paper/2205.12256/

## Key facts

- Cited by 39 papers in the map (data through 14 Oct 2026).
- Builds on 30 of its 62 references that are in the map.
- First posted on arXiv 24 May 2022.

## Builds on (most cited first, 30 of 30)

- [AMASS: Archive of Motion Capture as Surface Shapes](https://mlsanity.com/paper/1904.03278/) (2019)
- [End-to-end Recovery of Human Shape and Pose](https://mlsanity.com/paper/1712.06584/) (2017)
- [VIBE: Video Inference for Human Body Pose and Shape Estimation](https://mlsanity.com/paper/1912.05656/) (2019)
- [Brax -- A Differentiable Physics Engine for Large Scale Rigid Body Simulation](https://mlsanity.com/paper/2106.13281/) (2021)
- [HuMoR: 3D Human Motion Model for Robust Pose Estimation](https://mlsanity.com/paper/2105.04668/) (2021)
- [DiffTaichi: Differentiable Programming for Physical Simulation](https://mlsanity.com/paper/1910.00935/) (2019)
- [BEHAVE: Dataset and Method for Tracking Human Object Interactions](https://mlsanity.com/paper/2204.06950/) (2022)
- [Total Capture: A 3D Deformation Model for Tracking Faces, Hands, and Bodies](https://mlsanity.com/paper/1801.01615/) (2018)
- [Single-Shot Multi-Person 3D Pose Estimation From Monocular RGB](https://mlsanity.com/paper/1712.03453/) (2017)
- [Long-term Human Motion Prediction with Scene Context](https://mlsanity.com/paper/2007.03672/) (2020)
- [DeepMimic: Example-Guided Deep Reinforcement Learning of Physics-Based Character Skills](https://mlsanity.com/paper/1804.02717/) (2018)
- [Monocular Total Capture: Posing Face, Body, and Hands in the Wild](https://mlsanity.com/paper/1812.01598/) (2018)
- [Perceiving 3D Human-Object Spatial Arrangements from a Single Image in the Wild](https://mlsanity.com/paper/2007.15649/) (2020)
- [VNect: Real-time 3D Human Pose Estimation with a Single RGB Camera](https://mlsanity.com/paper/1705.01583/) (2017)
- [SimPoE: Simulated Character Control for 3D Human Pose Estimation](https://mlsanity.com/paper/2104.00683/) (2021)
- [Dynamics-Regulated Kinematic Policy for Egocentric Pose Estimation](https://mlsanity.com/paper/2106.05969/) (2021)
- [Contact and Human Dynamics from Monocular Video](https://mlsanity.com/paper/2007.11678/) (2020)
- [Exploiting temporal information for 3D pose estimation](https://mlsanity.com/paper/1711.08585/) (2017)
- [Gradients are Not All You Need](https://mlsanity.com/paper/2111.05803/) (2021)
- [NeuralSim: Augmenting Differentiable Simulators with Neural Networks](https://mlsanity.com/paper/2011.04217/) (2020)
- [Neural Monocular 3D Human Motion Capture with Physical Awareness](https://mlsanity.com/paper/2105.01057/) (2021)
- [Physics-based Human Motion Estimation and Synthesis from Videos](https://mlsanity.com/paper/2109.09913/) (2021)
- [Residual Force Control for Agile Human Behavior Imitation and Extended Motion Synthesis](https://mlsanity.com/paper/2006.07364/) (2020)
- [Neural Descent for Visual 3D Human Pose and Shape](https://mlsanity.com/paper/2008.06910/) (2020)
- [Fast and Feature-Complete Differentiable Physics for Articulated Rigid Bodies with Contact](https://mlsanity.com/paper/2103.16021/) (2021)
- [PhysCap: Physically Plausible Monocular 3D Motion Capture in Real Time](https://mlsanity.com/paper/2008.08880/) (2020)
- [THUNDR: Transformer-based 3D HUmaN Reconstruction with Markers](https://mlsanity.com/paper/2106.09336/) (2021)
- [Building Statistical Shape Spaces for 3D Human Modeling](https://mlsanity.com/paper/1503.05860/) (2015)
- [Efficient Differentiable Simulation of Articulated Bodies](https://mlsanity.com/paper/2109.07719/) (2021)
- [TransPose: Real-time 3D Human Translation and Pose Estimation with Six Inertial Sensors](https://mlsanity.com/paper/2105.04605/) (2021)

## Built on by (most cited first, 30 of 39)

- [EMAGE: Towards Unified Holistic Co-Speech Gesture Generation via Expressive Masked Audio Gesture Modeling](https://mlsanity.com/paper/2401.00374/) (2023)
- [3D Human Pose Estimation via Intuitive Physics](https://mlsanity.com/paper/2303.18246/) (2023)
- [DECO: Dense Estimation of 3D Human-Scene Contact In The Wild](https://mlsanity.com/paper/2309.15273/) (2023)
- [RoHM: Robust Human Motion Reconstruction via Diffusion](https://mlsanity.com/paper/2401.08570/) (2024)
- [AnySkill: Learning Open-Vocabulary Physical Skill for Interactive Agents](https://mlsanity.com/paper/2403.12835/) (2024)
- [Physics-Informed Computer Vision: A Review and Perspectives](https://mlsanity.com/paper/2305.18035/) (2023)
- [PhysPT: Physics-aware Pretrained Transformer for Estimating Human Dynamics from Monocular Videos](https://mlsanity.com/paper/2404.04430/) (2024)
- [MotionCraft: Crafting Whole-Body Motion with Plug-and-Play Multimodal Controls](https://mlsanity.com/paper/2407.21136/) (2024)
- [Decaf: Monocular Deformation Capture for Face and Hand Interactions](https://mlsanity.com/paper/2309.16670/) (2023)
- [MultiPhys: Multi-Person Physics-aware 3D Motion Estimation](https://mlsanity.com/paper/2404.11987/) (2024)
- [Contact-aware Human Motion Generation from Textual Descriptions](https://mlsanity.com/paper/2403.15709/) (2024)
- [Transformer-Based Learned Optimization](https://mlsanity.com/paper/2212.01055/) (2022)
- [ImDy: Human Inverse Dynamics from Imitated Observations](https://mlsanity.com/paper/2410.17610/) (2024)
- [FinePhys: Fine-grained Human Action Generation by Explicitly Incorporating Physical Laws for Effective Skeletal Guidance](https://mlsanity.com/paper/2505.13437/) (2025)
- [DeepSimHO: Stable Pose Estimation for Hand-Object Interaction via Physics Simulation](https://mlsanity.com/paper/2310.07206/) (2023)
- [Learned Neural Physics Simulation for Articulated 3D Human Pose Reconstruction](https://mlsanity.com/paper/2410.12023/) (2024)
- [PhysHMR: Learning Humanoid Control Policies from Vision for Physically Plausible Human Motion Reconstruction](https://mlsanity.com/paper/2510.02566/) (2025)
- [Optimal-state Dynamics Estimation for Physics-based Human Motion Capture from Videos](https://mlsanity.com/paper/2410.07795/) (2024)
- [Human Motion Prediction under Unexpected Perturbation](https://mlsanity.com/paper/2403.15891/) (2024)
- [A Plug-and-Play Physical Motion Restoration Approach for In-the-Wild High-Difficulty Motions](https://mlsanity.com/paper/2412.17377/) (2024)
- [GraMMaR: Ground-aware Motion Model for 3D Human Motion Reconstruction](https://mlsanity.com/paper/2306.16736/) (2023)
- [HandyPriors: Physically Consistent Perception of Hand-Object Interactions with Differentiable Priors](https://mlsanity.com/paper/2311.16552/) (2023)
- [QuaMo: Quaternion Motions for Vision-based 3D Human Kinematics Capture](https://mlsanity.com/paper/2601.19580/) (2026)
- [Physics-Guided Human Motion Capture with Pose Probability Modeling](https://mlsanity.com/paper/2308.09910/) (2023)
- [Bayesian Differentiable Physics for Cloth Digitalization](https://mlsanity.com/paper/2402.17664/) (2024)
- [Prof. Robot: Differentiable Robot Rendering Without Static and Self-Collisions](https://mlsanity.com/paper/2503.11269/) (2025)
- [Homogeneous Dynamics Space for Heterogeneous Humans](https://mlsanity.com/paper/2412.06146/) (2024)
- [Multi-person Physics-based Pose Estimation for Combat Sports](https://mlsanity.com/paper/2504.08175/) (2025)
- [Beyond Motion Pattern: An Empirical Study of Physical Forces for Human Motion Understanding](https://mlsanity.com/paper/2512.20451/) (2025)
- [GMT: Goal-Conditioned Multimodal Transformer for 6-DOF Object Trajectory Synthesis in 3D Scenes](https://mlsanity.com/paper/2603.17993/) (2026)

## Abstract (arXiv)

We introduce DiffPhy, a differentiable physics-based model for articulated 3d human motion reconstruction from video. Applications of physics-based reasoning in human motion analysis have so far been limited, both by the complexity of constructing adequate physical models of articulated human motion, and by the formidable challenges of performing stable and efficient inference with physics in the loop. We jointly address such modeling and inference challenges by proposing an approach that combines a physically plausible body representation with anatomical joint limits, a differentiable physics simulator, and optimization techniques that ensure good performance and robustness to suboptimal local optima. In contrast to several recent methods, our approach readily supports full-body contact including interactions with objects in the scene. Most importantly, our model connects end-to-end with images, thus supporting direct gradient-based physics optimization by means of image-based loss functions. We validate the model by demonstrating that it can accurately reconstruct physically plausible 3d human motion from monocular video, both on public benchmarks with available 3d ground-truth, and on videos from the internet.