# PhysHMR: Learning Humanoid Control Policies from Vision for Physically Plausible Human Motion Reconstruction

arXiv: https://arxiv.org/abs/2510.02566 · HTML: https://mlsanity.com/paper/2510.02566/

## Key facts

- Cited by 7 papers in the map (data through 14 Oct 2026).
- Builds on 35 of its 55 references that are in the map.
- First posted on arXiv 2 Oct 2025.

## Builds on (most cited first, 30 of 35)

- [Expressive Body Capture: 3D Hands, Face, and Body from a Single Image](https://mlsanity.com/paper/1904.05866/) (2019)
- [Isaac Gym: High Performance GPU-Based Physics Simulation For Robot Learning](https://mlsanity.com/paper/2108.10470/) (2021)
- [Keep it SMPL: Automatic Estimation of 3D Human Pose and Shape from a Single Image](https://mlsanity.com/paper/1607.08128/) (2016)
- [Neural Body: Implicit Neural Representations with Structured Latent Codes for Novel View Synthesis of Dynamic Humans](https://mlsanity.com/paper/2012.15838/) (2020)
- [Humans in 4D: Reconstructing and Tracking Humans with Transformers](https://mlsanity.com/paper/2305.20091/) (2023)
- [HuMoR: 3D Human Motion Model for Robust Pose Estimation](https://mlsanity.com/paper/2105.04668/) (2021)
- [CLIFF: Carrying Location Information in Full Frames into Human Pose and Shape Estimation](https://mlsanity.com/paper/2208.00571/) (2022)
- [Perpetual Humanoid Control for Real-time Simulated Avatars](https://mlsanity.com/paper/2305.06456/) (2023)
- [WHAM: Reconstructing World-grounded Humans with Accurate 3D Motion](https://mlsanity.com/paper/2312.07531/) (2023)
- [STAR: Sparse Trained Articulated Human Body Regressor](https://mlsanity.com/paper/2008.08535/) (2020)
- [DeepMimic: Example-Guided Deep Reinforcement Learning of Physics-Based Character Skills](https://mlsanity.com/paper/1804.02717/) (2018)
- [Monocular Total Capture: Posing Face, Body, and Hands in the Wild](https://mlsanity.com/paper/1812.01598/) (2018)
- [GLAMR: Global Occlusion-Aware Human Mesh Recovery with Dynamic Cameras](https://mlsanity.com/paper/2112.01524/) (2021)
- [Decoupling Human and Camera Motion from Videos in the Wild](https://mlsanity.com/paper/2302.12827/) (2023)
- [World-Grounded Human Motion Recovery via Gravity-View Coordinates](https://mlsanity.com/paper/2409.06662/) (2024)
- [Universal Humanoid Motion Representations for Physics-Based Control](https://mlsanity.com/paper/2310.04582/) (2023)
- [MaskedMimic: Unified Physics-Based Character Control Through Masked Motion Inpainting](https://mlsanity.com/paper/2409.14393/) (2024)
- [SimPoE: Simulated Character Control for 3D Human Pose Estimation](https://mlsanity.com/paper/2104.00683/) (2021)
- [Exploiting temporal context for 3D human pose estimation in the wild](https://mlsanity.com/paper/1905.04266/) (2019)
- [TRAM: Global Trajectory and Motion of 3D Humans from in-the-wild Videos](https://mlsanity.com/paper/2403.17346/) (2024)
- [3D Human Pose Estimation via Intuitive Physics](https://mlsanity.com/paper/2303.18246/) (2023)
- [EMDB: The Electromagnetic Database of Global 3D Human Pose and Shape in the Wild](https://mlsanity.com/paper/2308.16894/) (2023)
- [CALM: Conditional Adversarial Latent Models for Directable Virtual Characters](https://mlsanity.com/paper/2305.02195/) (2023)
- [TRACE: 5D Temporal Regression of Avatars with Dynamic Cameras in 3D Environments](https://mlsanity.com/paper/2306.02850/) (2023)
- [QuestSim: Human Motion Tracking from Sparse Sensors with Simulated Avatars](https://mlsanity.com/paper/2209.09391/) (2022)
- [PhysCap: Physically Plausible Monocular 3D Motion Capture in Real Time](https://mlsanity.com/paper/2008.08880/) (2020)
- [D&D: Learning Human Dynamics from Dynamic Camera](https://mlsanity.com/paper/2209.08790/) (2022)
- [Differentiable Dynamics for Articulated 3d Human Motion Reconstruction](https://mlsanity.com/paper/2205.12256/) (2022)
- [SMPLest-X: Ultimate Scaling for Expressive Human Pose and Shape Estimation](https://mlsanity.com/paper/2501.09782/) (2025)
- [PhysPT: Physics-aware Pretrained Transformer for Estimating Human Dynamics from Monocular Videos](https://mlsanity.com/paper/2404.04430/) (2024)

## Built on by (most cited first, 7 of 7)

- [DuoMo: Dual Motion Diffusion for World-Space Human Reconstruction](https://mlsanity.com/paper/2603.03265/) (2026)
- [EchoAvatar: Real-time Generative Avatar Animation from Audio Streams](https://mlsanity.com/paper/2605.28272/) (2026)
- [HandFlow: Fully Generative 4D Hand Recovery with Flow Matching](https://mlsanity.com/paper/2607.11221/) (2026)
- [ResiHMR: Residual-Limb Aware Single-Image 3D Human Mesh Recovery for Individuals with Limb Loss](https://mlsanity.com/paper/2604.28025/) (2026)
- [Recovering Physically Plausible Human-Object Interactions from Monocular Videos](https://mlsanity.com/paper/2606.05359/) (2026)
- [FlowHMR: Physically Plausible Motion Capture from Video](https://mlsanity.com/paper/2610.03691/) (2026)
- [A Diagnostic Study of World-Space Human Recovery Under High Camera Dynamics](https://mlsanity.com/paper/2610.19729/) (2026)

## Abstract (arXiv)

Reconstructing physically plausible human motion from monocular videos remains a challenging problem in computer vision and graphics. Existing methods primarily focus on kinematics-based pose estimation, often leading to unrealistic results due to the lack of physical constraints. To address such artifacts, prior methods have typically relied on physics-based post-processing following the initial kinematics-based motion estimation. However, this two-stage design introduces error accumulation, ultimately limiting the overall reconstruction quality. In this paper, we present PhysHMR, a unified framework that directly learns a visual-to-action policy for humanoid control in a physics-based simulator, enabling motion reconstruction that is both physically grounded and visually aligned with the input video. A key component of our approach is the pixel-as-ray strategy, which lifts 2D keypoints into 3D spatial rays and transforms them into global space. These rays are incorporated as policy inputs, providing robust global pose guidance without depending on noisy 3D root predictions. This soft global grounding, combined with local visual features from a pretrained encoder, allows the policy to reason over both detailed pose and global positioning. To overcome the sample inefficiency of reinforcement learning, we further introduce a distillation scheme that transfers motion knowledge from a mocap-trained expert to the vision-conditioned policy, which is then refined using physically motivated reinforcement learning rewards. Extensive experiments demonstrate that PhysHMR produces high-fidelity, physically plausible motion across diverse scenarios, outperforming prior approaches in both visual accuracy and physical realism.