# xR-EgoPose: Egocentric 3D Human Pose from an HMD Camera

arXiv: https://arxiv.org/abs/1907.10045 · HTML: https://mlsanity.com/paper/1907.10045/

## Key facts

- Cited by 79 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 23 Jul 2019.

## Built on by (most cited first, 30 of 79)

- [Ego-Exo4D: Understanding Skilled Human Activity from First- and Third-Person Perspectives](https://mlsanity.com/paper/2311.18259/) (2023)
- [BEDLAM: A Synthetic Dataset of Bodies Exhibiting Detailed Lifelike Animated Motion](https://mlsanity.com/paper/2306.16940/) (2023)
- [EgoBody: Human Body Shape and Motion of Interacting People from Head-Mounted Devices](https://mlsanity.com/paper/2112.07642/) (2021)
- [Dynamics-Regulated Kinematic Policy for Egocentric Pose Estimation](https://mlsanity.com/paper/2106.05969/) (2021)
- [Ego-Body Pose Estimation via Ego-Head Pose Estimation](https://mlsanity.com/paper/2212.04636/) (2022)
- [An Outlook into the Future of Egocentric Vision](https://mlsanity.com/paper/2308.07123/) (2023)
- [GIMO: Gaze-Informed Human Motion Prediction in Context](https://mlsanity.com/paper/2204.09443/) (2022)
- [Estimating Body and Hand Motion in an Ego-sensed World](https://mlsanity.com/paper/2410.03665/) (2024)
- [UnrealEgo: A New Dataset for Robust Egocentric 3D Human Motion Capture](https://mlsanity.com/paper/2208.01633/) (2022)
- [Estimating Egocentric 3D Human Pose in Global Space](https://mlsanity.com/paper/2104.13454/) (2021)
- [EgoLocate: Real-time Motion Capture, Localization, and Mapping with Sparse Body-mounted Sensors](https://mlsanity.com/paper/2305.01599/) (2023)
- [EgoObjects: A Large-Scale Egocentric Dataset for Fine-Grained Object Understanding](https://mlsanity.com/paper/2309.08816/) (2023)
- [Estimating Egocentric 3D Human Pose in the Wild with External Weak Supervision](https://mlsanity.com/paper/2201.07929/) (2022)
- [Probabilistic Human Mesh Recovery in 3D Scenes from Egocentric Views](https://mlsanity.com/paper/2304.06024/) (2023)
- [Egocentric Whole-Body Motion Capture with FisheyeViT and Diffusion-Based Motion Refinement](https://mlsanity.com/paper/2311.16495/) (2023)
- [Mocap Everyone Everywhere: Lightweight Motion Capture With Smartwatches and a Head-Mounted Camera](https://mlsanity.com/paper/2401.00847/) (2024)
- [Scene-aware Egocentric 3D Human Pose Estimation](https://mlsanity.com/paper/2212.11684/) (2022)
- [EventEgo3D: 3D Human Motion Capture from Egocentric Event Streams](https://mlsanity.com/paper/2404.08640/) (2024)
- [Egocentric Pose Estimation from Human Vision Span](https://mlsanity.com/paper/2104.05167/) (2021)
- [EgoGen: An Egocentric Synthetic Data Generator](https://mlsanity.com/paper/2401.08739/) (2024)
- [EgoLoc: Revisiting 3D Object Localization from Egocentric Videos with Visual Queries](https://mlsanity.com/paper/2212.06969/) (2022)
- [BEDLAM2.0: Synthetic Humans and Cameras in Motion](https://mlsanity.com/paper/2511.14394/) (2025)
- [3D Segmentation of Humans in Point Clouds with Synthetic Data](https://mlsanity.com/paper/2212.00786/) (2022)
- [3D Human Pose Perception from Egocentric Stereo Videos](https://mlsanity.com/paper/2401.00889/) (2023)
- [Ego3DPose: Capturing 3D Cues from Binocular Egocentric Views](https://mlsanity.com/paper/2309.11962/) (2023)
- [Reconstructing Humans with a Biomechanically Accurate Skeleton](https://mlsanity.com/paper/2503.21751/) (2025)
- [Challenges and Trends in Egocentric Vision: A Survey](https://mlsanity.com/paper/2503.15275/) (2025)
- [Revisiting 3D Object Detection From an Egocentric Perspective](https://mlsanity.com/paper/2112.07787/) (2021)
- [Ultra Inertial Poser: Scalable Motion Capture and Tracking from Sparse Inertial Sensors and Ultra-Wideband Ranging](https://mlsanity.com/paper/2404.19541/) (2024)
- [Ego4o: Egocentric Human Motion Capture and Understanding from Multi-Modal Input](https://mlsanity.com/paper/2504.08449/) (2025)

## Abstract (arXiv)

We present a new solution to egocentric 3D body pose estimation from monocular images captured from a downward looking fish-eye camera installed on the rim of a head mounted virtual reality device. This unusual viewpoint, just 2 cm. away from the user's face, leads to images with unique visual appearance, characterized by severe self-occlusions and strong perspective distortions that result in a drastic difference in resolution between lower and upper body. Our contribution is two-fold. Firstly, we propose a new encoder-decoder architecture with a novel dual branch decoder designed specifically to account for the varying uncertainty in the 2D joint locations. Our quantitative evaluation, both on synthetic and real-world datasets, shows that our strategy leads to substantial improvements in accuracy over state of the art egocentric pose estimation approaches. Our second contribution is a new large-scale photorealistic synthetic dataset - xR-EgoPose - offering 383K frames of high quality renderings of people with a diversity of skin tones, body shapes, clothing, in a variety of backgrounds and lighting conditions, performing a range of actions. Our experiments show that the high variability in our new synthetic training corpus leads to good generalization to real world footage and to state of the art results on real world datasets with ground truth. Moreover, an evaluation on the Human3.6M benchmark shows that the performance of our method is on par with top performing approaches on the more classic problem of 3D human pose from a third person viewpoint.