# MultiPhys: Multi-Person Physics-aware 3D Motion Estimation

arXiv: https://arxiv.org/abs/2404.11987 · HTML: https://mlsanity.com/paper/2404.11987/

## Key facts

- Cited by 18 papers in the map (data through 14 Oct 2026).
- Builds on 33 of its 49 references that are in the map.
- First posted on arXiv 18 Apr 2024.

## Builds on (most cited first, 30 of 33)

- [Proximal Policy Optimization Algorithms](https://mlsanity.com/paper/1707.06347/) (2017)
- [Expressive Body Capture: 3D Hands, Face, and Body from a Single Image](https://mlsanity.com/paper/1904.05866/) (2019)
- [AMASS: Archive of Motion Capture as Surface Shapes](https://mlsanity.com/paper/1904.03278/) (2019)
- [End-to-end Recovery of Human Shape and Pose](https://mlsanity.com/paper/1712.06584/) (2017)
- [VIBE: Video Inference for Human Body Pose and Shape Estimation](https://mlsanity.com/paper/1912.05656/) (2019)
- [Humans in 4D: Reconstructing and Tracking Humans with Transformers](https://mlsanity.com/paper/2305.20091/) (2023)
- [HuMoR: 3D Human Motion Model for Robust Pose Estimation](https://mlsanity.com/paper/2105.04668/) (2021)
- [Learning 3D Human Dynamics from Video](https://mlsanity.com/paper/1812.01601/) (2018)
- [Monocular, One-stage, Regression of Multiple 3D People](https://mlsanity.com/paper/2008.12272/) (2020)
- [Single-Shot Multi-Person 3D Pose Estimation From Monocular RGB](https://mlsanity.com/paper/1712.03453/) (2017)
- [Beyond Static Features for Temporally Consistent 3D Human Pose and Shape from a Video](https://mlsanity.com/paper/2011.08627/) (2020)
- [GLAMR: Global Occlusion-Aware Human Mesh Recovery with Dynamic Cameras](https://mlsanity.com/paper/2112.01524/) (2021)
- [Putting People in their Place: Monocular Regression of 3D People in Depth](https://mlsanity.com/paper/2112.08274/) (2021)
- [Decoupling Human and Camera Motion from Videos in the Wild](https://mlsanity.com/paper/2302.12827/) (2023)
- [Coherent Reconstruction of Multiple Humans from a Single Image](https://mlsanity.com/paper/2006.08586/) (2020)
- [SimPoE: Simulated Character Control for 3D Human Pose Estimation](https://mlsanity.com/paper/2104.00683/) (2021)
- [AvatarPoser: Articulated Full-Body Pose Tracking from Sparse Motion Sensing](https://mlsanity.com/paper/2207.13784/) (2022)
- [Dynamics-Regulated Kinematic Policy for Egocentric Pose Estimation](https://mlsanity.com/paper/2106.05969/) (2021)
- [3D Human Motion Estimation via Motion Compression and Refinement](https://mlsanity.com/paper/2008.03789/) (2020)
- [TRACE: 5D Temporal Regression of Avatars with Dynamic Cameras in 3D Environments](https://mlsanity.com/paper/2306.02850/) (2023)
- [QuestSim: Human Motion Tracking from Sparse Sensors with Simulated Avatars](https://mlsanity.com/paper/2209.09391/) (2022)
- [Multi-Person Extreme Motion Prediction](https://mlsanity.com/paper/2105.08825/) (2021)
- [Neural Monocular 3D Human Motion Capture with Physical Awareness](https://mlsanity.com/paper/2105.01057/) (2021)
- [Residual Force Control for Agile Human Behavior Imitation and Extended Motion Synthesis](https://mlsanity.com/paper/2006.07364/) (2020)
- [Occluded Human Mesh Recovery](https://mlsanity.com/paper/2203.13349/) (2022)
- [Human Mesh Recovery from Multiple Shots](https://mlsanity.com/paper/2012.09843/) (2020)
- [Generative Proxemics: A Prior for 3D Social Interaction from Images](https://mlsanity.com/paper/2306.09337/) (2023)
- [Realistic Full-Body Tracking from Sparse Observations via Joint-Level Modeling](https://mlsanity.com/paper/2308.08855/) (2023)
- [D&D: Learning Human Dynamics from Dynamic Camera](https://mlsanity.com/paper/2209.08790/) (2022)
- [Differentiable Dynamics for Articulated 3d Human Motion Reconstruction](https://mlsanity.com/paper/2205.12256/) (2022)

## Built on by (most cited first, 18 of 18)

- [Visual Imitation Enables Contextual Humanoid Control](https://mlsanity.com/paper/2505.03729/) (2025)
- [Reconstructing Humans with a Biomechanically Accurate Skeleton](https://mlsanity.com/paper/2503.21751/) (2025)
- [Reconstructing 4D Spatial Intelligence: A Survey](https://mlsanity.com/paper/2507.21045/) (2025)
- [RAM: Recover Any 3D Human Motion in-the-Wild](https://mlsanity.com/paper/2603.19929/) (2026)
- [Ponimator: Unfolding Interactive Pose for Versatile Human-human Interaction Animation](https://mlsanity.com/paper/2510.14976/) (2025)
- [PhysiInter: Integrating Physical Mapping for High-Fidelity Human Interaction Generation](https://mlsanity.com/paper/2506.07456/) (2025)
- [Reconstructing Close Human Interaction with Appearance and Proxemics Reasoning](https://mlsanity.com/paper/2507.02565/) (2025)
- [PP-Motion: Physical-Perceptual Fidelity Evaluation for Human Motion Generation](https://mlsanity.com/paper/2508.08179/) (2025)
- [Group Inertial Poser: Multi-Person Pose and Global Translation from Sparse Inertial Sensors and Ultra-Wideband Ranging](https://mlsanity.com/paper/2510.21654/) (2025)
- [HumanScore: Benchmarking Human Motions in Generated Videos](https://mlsanity.com/paper/2604.20157/) (2026)
- [Multi-person Physics-based Pose Estimation for Combat Sports](https://mlsanity.com/paper/2504.08175/) (2025)
- [EgoExoMoCap: Distributed Ego-Exo Human Motion Capture](https://mlsanity.com/paper/2607.15868/) (2026)
- [SocialMirror: Reconstructing 3D Human Interaction Behaviors from Monocular Videos with Semantic and Geometric Guidance](https://mlsanity.com/paper/2604.13581/) (2026)
- [Recovering Physically Plausible Human-Object Interactions from Monocular Videos](https://mlsanity.com/paper/2606.05359/) (2026)
- [TaskNPoint: How to Teach Your Humanoid to Hit a Backhand in Minutes](https://mlsanity.com/paper/2606.26215/) (2026)
- [GNOCHI: Generative Neural mOdel for Close Human-Human Interactions](https://mlsanity.com/paper/2607.10408/) (2026)
- [Inter-X++: A Comprehensive Benchmark for Multimodal Human-Human Interaction Analysis](https://mlsanity.com/paper/2608.20312/) (2026)
- [WeLike2Party! In-Context Motion Transfer for Multi-Human Image Animation](https://mlsanity.com/paper/2609.36937/) (2026)

## Abstract (arXiv)

We introduce MultiPhys, a method designed for recovering multi-person motion from monocular videos. Our focus lies in capturing coherent spatial placement between pairs of individuals across varying degrees of engagement. MultiPhys, being physically aware, exhibits robustness to jittering and occlusions, and effectively eliminates penetration issues between the two individuals. We devise a pipeline in which the motion estimated by a kinematic-based method is fed into a physics simulator in an autoregressive manner. We introduce distinct components that enable our model to harness the simulator's properties without compromising the accuracy of the kinematic estimates. This results in final motion estimates that are both kinematically coherent and physically compliant. Extensive evaluations on three challenging datasets characterized by substantial inter-person interaction show that our method significantly reduces errors associated with penetration and foot skating, while performing competitively with the state-of-the-art on motion accuracy and smoothness. Results and code can be found on our project page (http://www.iri.upc.edu/people/nugrinovic/multiphys/).