# WHAC: World-grounded Humans and Cameras

arXiv: https://arxiv.org/abs/2403.12959 · HTML: https://mlsanity.com/paper/2403.12959/

## Key facts

- Cited by 30 papers in the map (data through 14 Oct 2026).
- Builds on 39 of its 51 references that are in the map.
- First posted on arXiv 19 Mar 2024.

## Builds on (most cited first, 30 of 39)

- [Learning Phrase Representations using RNN Encoder-Decoder for Statistical Machine Translation](https://mlsanity.com/paper/1406.1078/) (2014)
- [Expressive Body Capture: 3D Hands, Face, and Body from a Single Image](https://mlsanity.com/paper/1904.05866/) (2019)
- [AMASS: Archive of Motion Capture as Surface Shapes](https://mlsanity.com/paper/1904.03278/) (2019)
- [DROID-SLAM: Deep Visual SLAM for Monocular, Stereo, and RGB-D Cameras](https://mlsanity.com/paper/2108.10869/) (2021)
- [Neural Body: Implicit Neural Representations with Structured Latent Codes for Novel View Synthesis of Dynamic Humans](https://mlsanity.com/paper/2012.15838/) (2020)
- [HuMoR: 3D Human Motion Model for Robust Pose Estimation](https://mlsanity.com/paper/2105.04668/) (2021)
- [BEDLAM: A Synthetic Dataset of Bodies Exhibiting Detailed Lifelike Animated Motion](https://mlsanity.com/paper/2306.16940/) (2023)
- [WHAM: Reconstructing World-grounded Humans with Accurate 3D Motion](https://mlsanity.com/paper/2312.07531/) (2023)
- [Deep Patch Visual Odometry](https://mlsanity.com/paper/2208.04726/) (2022)
- [FrankMocap: A Monocular 3D Whole-Body Pose Estimation System via Regression and Integration](https://mlsanity.com/paper/2108.06428/) (2021)
- [Monocular Expressive Body Regression through Body-Driven Attention](https://mlsanity.com/paper/2008.09062/) (2020)
- [SMPLer-X: Scaling Up Expressive Human Pose and Shape Estimation](https://mlsanity.com/paper/2309.17448/) (2023)
- [GLAMR: Global Occlusion-Aware Human Mesh Recovery with Dynamic Cameras](https://mlsanity.com/paper/2112.01524/) (2021)
- [PyMAF-X: Towards Well-aligned Full-body Model Regression from Monocular Images](https://mlsanity.com/paper/2207.06400/) (2022)
- [Decoupling Human and Camera Motion from Videos in the Wild](https://mlsanity.com/paper/2302.12827/) (2023)
- [EgoBody: Human Body Shape and Motion of Interacting People from Head-Mounted Devices](https://mlsanity.com/paper/2112.07642/) (2021)
- [Human POSEitioning System (HPS): 3D Human Pose Estimation and Self-localization in Large Scenes from Body-Mounted Sensors](https://mlsanity.com/paper/2103.17265/) (2021)
- [Collaborative Regression of Expressive Bodies using Moderation](https://mlsanity.com/paper/2105.05301/) (2021)
- [One-Stage 3D Whole-Body Mesh Recovery with Component Aware Transformer](https://mlsanity.com/paper/2303.16160/) (2023)
- [HuMMan: Multi-Modal 4D Human Dataset for Versatile Sensing and Modeling](https://mlsanity.com/paper/2204.13686/) (2022)
- [DNA-Rendering: A Diverse Neural Actor Repository for High-Fidelity Human-centric Rendering](https://mlsanity.com/paper/2307.10173/) (2023)
- [EMDB: The Electromagnetic Database of Global 3D Human Pose and Shape in the Wild](https://mlsanity.com/paper/2308.16894/) (2023)
- [TRACE: 5D Temporal Regression of Avatars with Dynamic Cameras in 3D Environments](https://mlsanity.com/paper/2306.02850/) (2023)
- [Accurate 3D Hand Pose Estimation for Whole-Body 3D Human Mesh Estimation](https://mlsanity.com/paper/2011.11534/) (2020)
- [Playing for 3D Human Recovery](https://mlsanity.com/paper/2110.07588/) (2021)
- [SynBody: Synthetic Dataset with Layered Human Models for 3D Human Perception and Modeling](https://mlsanity.com/paper/2303.17368/) (2023)
- [Monocular Real-time Full Body Capture with Inter-part Correlations](https://mlsanity.com/paper/2012.06087/) (2020)
- [4D Association Graph for Realtime Multi-person Motion Capture Using Multiple Video Cameras](https://mlsanity.com/paper/2002.12625/) (2020)
- [HybrIK-X: Hybrid Analytical-Neural Inverse Kinematics for Whole-body Mesh Recovery](https://mlsanity.com/paper/2304.05690/) (2023)
- [4D Human Body Capture from Egocentric Video via 3D Scene Grounding](https://mlsanity.com/paper/2011.13341/) (2020)

## Built on by (most cited first, 30 of 30)

- [World-Grounded Human Motion Recovery via Gravity-View Coordinates](https://mlsanity.com/paper/2409.06662/) (2024)
- [Deep Patch Visual SLAM](https://mlsanity.com/paper/2408.01654/) (2024)
- [HaWoR: World-Space Hand Motion Reconstruction from Egocentric Videos](https://mlsanity.com/paper/2501.02973/) (2025)
- [SMPLest-X: Ultimate Scaling for Expressive Human Pose and Shape Estimation](https://mlsanity.com/paper/2501.09782/) (2025)
- [UniTalker: Scaling up Audio-Driven 3D Facial Animation through A Unified Model](https://mlsanity.com/paper/2408.00762/) (2024)
- [Joint Optimization for 4D Human-Scene Reconstruction in the Wild](https://mlsanity.com/paper/2501.02158/) (2025)
- [ZeroHSI: Zero-Shot 4D Human-Scene Interaction by Video Generation](https://mlsanity.com/paper/2412.18600/) (2024)
- [BEDLAM2.0: Synthetic Humans and Cameras in Motion](https://mlsanity.com/paper/2511.14394/) (2025)
- [Free-Form Motion Control: Controlling the 6D Poses of Camera and Objects in Video Generation](https://mlsanity.com/paper/2501.01425/) (2025)
- [CoMotion: Concurrent Multi-person 3D Motion](https://mlsanity.com/paper/2504.12186/) (2025)
- [DanceTogether! Identity-Preserving Multi-Person Interactive Video Generation](https://mlsanity.com/paper/2505.18078/) (2025)
- [InteractAnything: Zero-shot Human Object Interaction Synthesis via LLM Feedback and Object Affordance Parsing](https://mlsanity.com/paper/2505.24315/) (2025)
- [RealisMotion: Decomposed Human Motion Control and Video Generation in the World Space](https://mlsanity.com/paper/2508.08588/) (2025)
- [Human Mesh Modeling for Anny Body](https://mlsanity.com/paper/2511.03589/) (2025)
- [MoCapAnything: Unified 3D Motion Capture for Arbitrary Skeletons from Monocular Videos](https://mlsanity.com/paper/2512.10881/) (2025)
- [ADHMR: Aligning Diffusion-based Human Mesh Recovery via Direct Preference Optimization](https://mlsanity.com/paper/2505.10250/) (2025)
- [S2R-HDR: A Large-Scale Rendered Dataset for HDR Fusion](https://mlsanity.com/paper/2504.07667/) (2025)
- [DiffProxy: Multi-View Human Mesh Recovery via Diffusion-Generated Dense Proxies](https://mlsanity.com/paper/2601.02267/) (2026)
- [JRDB-Pose3D: A Multi-person 3D Human Pose and Shape Estimation Dataset for Robotics](https://mlsanity.com/paper/2602.03064/) (2026)
- [LAMP: Localization Aware Multi-camera People Tracking in Metric 3D World](https://mlsanity.com/paper/2605.05390/) (2026)
- [DanceHMR: Hand-Aware Whole-Body Human Mesh Recovery from Monocular Videos](https://mlsanity.com/paper/2605.18102/) (2026)
- [MAMMA: Markerless & Automatic Multi-Person Motion Action Capture](https://mlsanity.com/paper/2506.13040/) (2025)
- [FantasyHSI: Video-Generation-Centric 4D Human Synthesis In Any Scene through A Graph-based Multi-Agent Framework](https://mlsanity.com/paper/2509.01232/) (2025)
- [WATCH: World-aware Allied Trajectory and pose reConstruction for Camera and Human](https://mlsanity.com/paper/2509.04600/) (2025)
- [Robust Motion Generation using Part-level Reliable Data from Videos](https://mlsanity.com/paper/2512.12703/) (2025)
- [CRISP: Contact-Guided Real2Sim from Monocular Video with Planar Scene Primitives](https://mlsanity.com/paper/2512.14696/) (2025)
- [Improving 3D Foot Motion Reconstruction in Markerless Monocular Human Motion Capture](https://mlsanity.com/paper/2603.09681/) (2026)
- [Natural Human Motion Recovery by Aligning High-Order Temporal Dynamics from Monocular Videos](https://mlsanity.com/paper/2605.26879/) (2026)
- [RoboSTAR: Next-Scale Autoregressive Sign Language Translation for Humanoid Robots](https://mlsanity.com/paper/2609.32250/) (2026)
- [A Diagnostic Study of World-Space Human Recovery Under High Camera Dynamics](https://mlsanity.com/paper/2610.19729/) (2026)

## Abstract (arXiv)

Estimating human and camera trajectories with accurate scale in the world coordinate system from a monocular video is a highly desirable yet challenging and ill-posed problem. In this study, we aim to recover expressive parametric human models (i.e., SMPL-X) and corresponding camera poses jointly, by leveraging the synergy between three critical players: the world, the human, and the camera. Our approach is founded on two key observations. Firstly, camera-frame SMPL-X estimation methods readily recover absolute human depth. Secondly, human motions inherently provide absolute spatial cues. By integrating these insights, we introduce a novel framework, referred to as WHAC, to facilitate world-grounded expressive human pose and shape estimation (EHPS) alongside camera pose estimation, without relying on traditional optimization techniques. Additionally, we present a new synthetic dataset, WHAC-A-Mole, which includes accurately annotated humans and cameras, and features diverse interactive human motions as well as realistic camera trajectories. Extensive experiments on both standard and newly established benchmarks highlight the superiority and efficacy of our framework. We will make the code and dataset publicly available.