# 4D Human Body Capture from Egocentric Video via 3D Scene Grounding

arXiv: https://arxiv.org/abs/2011.13341 · HTML: https://mlsanity.com/paper/2011.13341/

## Key facts

- Cited by 43 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 26 Nov 2020.

## Built on by (most cited first, 30 of 43)

- [HuMoR: 3D Human Motion Model for Robust Pose Estimation](https://mlsanity.com/paper/2105.04668/) (2021)
- [WHAM: Reconstructing World-grounded Humans with Accurate 3D Motion](https://mlsanity.com/paper/2312.07531/) (2023)
- [GLAMR: Global Occlusion-Aware Human Mesh Recovery with Dynamic Cameras](https://mlsanity.com/paper/2112.01524/) (2021)
- [Decoupling Human and Camera Motion from Videos in the Wild](https://mlsanity.com/paper/2302.12827/) (2023)
- [EgoBody: Human Body Shape and Motion of Interacting People from Head-Mounted Devices](https://mlsanity.com/paper/2112.07642/) (2021)
- [SPEC: Seeing People in the Wild with an Estimated Camera](https://mlsanity.com/paper/2110.00620/) (2021)
- [TRAM: Global Trajectory and Motion of 3D Humans from in-the-wild Videos](https://mlsanity.com/paper/2403.17346/) (2024)
- [Recovering 3D Human Mesh from Monocular Images: A Survey](https://mlsanity.com/paper/2203.01923/) (2022)
- [Learning Motion Priors for 4D Human Body Capture in 3D Scenes](https://mlsanity.com/paper/2108.10399/) (2021)
- [TRACE: 5D Temporal Regression of Avatars with Dynamic Cameras in 3D Environments](https://mlsanity.com/paper/2306.02850/) (2023)
- [An Outlook into the Future of Egocentric Vision](https://mlsanity.com/paper/2308.07123/) (2023)
- [Physics-based Human Motion Estimation and Synthesis from Videos](https://mlsanity.com/paper/2109.09913/) (2021)
- [The Wanderings of Odysseus in 3D Scenes](https://mlsanity.com/paper/2112.09251/) (2021)
- [In the Eye of Transformer: Global-Local Correlation for Egocentric Gaze Estimation](https://mlsanity.com/paper/2208.04464/) (2022)
- [EgoLocate: Real-time Motion Capture, Localization, and Mapping with Sparse Body-mounted Sensors](https://mlsanity.com/paper/2305.01599/) (2023)
- [SLOPER4D: A Scene-Aware Dataset for Global 4D Human Pose Estimation in Urban Environments](https://mlsanity.com/paper/2303.09095/) (2023)
- [CORE4D: A 4D Human-Object-Human Interaction Dataset for Collaborative Object REarrangement](https://mlsanity.com/paper/2406.19353/) (2024)
- [Neural Free-Viewpoint Performance Rendering under Complex Human-object Interactions](https://mlsanity.com/paper/2108.00362/) (2021)
- [PACE: Human and Camera Motion Estimation from in-the-wild Videos](https://mlsanity.com/paper/2310.13768/) (2023)
- [Probabilistic Human Mesh Recovery in 3D Scenes from Egocentric Views](https://mlsanity.com/paper/2304.06024/) (2023)
- [Reconstructing People, Places, and Cameras](https://mlsanity.com/paper/2412.17806/) (2024)
- [The One Where They Reconstructed 3D Humans and Environments in TV Shows](https://mlsanity.com/paper/2207.14279/) (2022)
- [WHAC: World-grounded Humans and Cameras](https://mlsanity.com/paper/2403.12959/) (2024)
- [Joint Optimization for 4D Human-Scene Reconstruction in the Wild](https://mlsanity.com/paper/2501.02158/) (2025)
- [CIMI4D: A Large Multimodal Climbing Motion Dataset under Human-scene Interactions](https://mlsanity.com/paper/2303.17948/) (2023)
- [EgoGen: An Egocentric Synthetic Data Generator](https://mlsanity.com/paper/2401.08739/) (2024)
- [ROAM: Robust and Object-Aware Motion Generation Using Neural Pose Descriptors](https://mlsanity.com/paper/2308.12969/) (2023)
- [Scene-Aware 3D Multi-Human Motion Capture from a Single Camera](https://mlsanity.com/paper/2301.05175/) (2023)
- [Synergistic Global-space Camera and Human Reconstruction from Videos](https://mlsanity.com/paper/2405.14855/) (2024)
- [COIN: Control-Inpainting Diffusion Prior for Human and Camera Motion Estimation](https://mlsanity.com/paper/2408.16426/) (2024)

## Abstract (arXiv)

We introduce a novel task of reconstructing a time series of second-person 3D human body meshes from monocular egocentric videos. The unique viewpoint and rapid embodied camera motion of egocentric videos raise additional technical barriers for human body capture. To address those challenges, we propose a simple yet effective optimization-based approach that leverages 2D observations of the entire video sequence and human-scene interaction constraint to estimate second-person human poses, shapes, and global motion that are grounded on the 3D environment captured from the egocentric view. We conduct detailed ablation studies to validate our design choice. Moreover, we compare our method with the previous state-of-the-art method on human motion capture from monocular video, and show that our method estimates more accurate human-body poses and shapes under the challenging egocentric setting. In addition, we demonstrate that our approach produces more realistic human-scene interaction.