# Ego3DPose: Capturing 3D Cues from Binocular Egocentric Views

arXiv: https://arxiv.org/abs/2309.11962 · HTML: https://mlsanity.com/paper/2309.11962/

## Key facts

- Cited by 22 papers in the map (data through 14 Oct 2026).
- Builds on 27 of its 46 references that are in the map.
- First posted on arXiv 21 Sep 2023.

## Builds on (most cited first, 27 of 27)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [PyTorch: An Imperative Style, High-Performance Deep Learning Library](https://mlsanity.com/paper/1912.01703/) (2019)
- [U-Net: Convolutional Networks for Biomedical Image Segmentation](https://mlsanity.com/paper/1505.04597/) (2015)
- [Efficient Object Localization Using Convolutional Networks](https://mlsanity.com/paper/1411.4280/) (2014)
- [Learning to Estimate 3D Human Pose and Shape from a Single Color Image](https://mlsanity.com/paper/1805.04092/) (2018)
- [Coarse-to-Fine Volumetric Prediction for Single-Image 3D Human Pose](https://mlsanity.com/paper/1611.07828/) (2016)
- [Learnable Triangulation of Human Pose](https://mlsanity.com/paper/1905.05754/) (2019)
- [Compositional Human Pose Regression](https://mlsanity.com/paper/1704.00159/) (2017)
- [3D Human Pose Estimation = 2D Pose Estimation + Matching](https://mlsanity.com/paper/1612.06524/) (2016)
- [Learning-Rate-Free Learning by D-Adaptation](https://mlsanity.com/paper/2301.07733/) (2023)
- [Generating Multiple Hypotheses for 3D Human Pose Estimation with Mixture Density Network](https://mlsanity.com/paper/1904.05547/) (2019)
- [Cross View Fusion for 3D Human Pose Estimation](https://mlsanity.com/paper/1909.01203/) (2019)
- [xR-EgoPose: Egocentric 3D Human Pose from an HMD Camera](https://mlsanity.com/paper/1907.10045/) (2019)
- [3D Human Pose Estimation from a Single Image via Distance Matrix Regression](https://mlsanity.com/paper/1611.09010/) (2016)
- [Mo2Cap2: Real-time Mobile 3D Motion Capture with a Cap-mounted Fisheye Camera](https://mlsanity.com/paper/1803.05959/) (2018)
- [UnrealEgo: A New Dataset for Robust Egocentric 3D Human Motion Capture](https://mlsanity.com/paper/2208.01633/) (2022)
- [Estimating Egocentric 3D Human Pose in Global Space](https://mlsanity.com/paper/2104.13454/) (2021)
- [Human pose estimation via Convolutional Part Heatmap Regression](https://mlsanity.com/paper/1609.01743/) (2016)
- [SelfPose: 3D Egocentric Pose Estimation from a Headset Mounted Camera](https://mlsanity.com/paper/2011.01519/) (2020)
- [Structured Prediction of 3D Human Pose with Deep Neural Networks](https://mlsanity.com/paper/1605.05180/) (2016)
- [AdaFuse: Adaptive Multiview Fusion for Accurate Human Pose Estimation in the Wild](https://mlsanity.com/paper/2010.13302/) (2020)
- [Maximum-Margin Structured Learning with Deep Networks for 3D Human Pose Estimation](https://mlsanity.com/paper/1508.06708/) (2015)
- [Generating Multiple Diverse Hypotheses for Human 3D Pose Consistent with 2D Joint Detections](https://mlsanity.com/paper/1702.02258/) (2017)
- [EgoCap: Egocentric Marker-less Motion Capture with Two Fisheye Cameras (Extended Abstract)](https://mlsanity.com/paper/1701.00142/) (2016)
- [Generalizable Human Pose Triangulation](https://mlsanity.com/paper/2110.00280/) (2021)
- [HEMlets PoSh: Learning Part-Centric Heatmap Triplets for 3D Human Pose and Shape Estimation](https://mlsanity.com/paper/2003.04894/) (2020)
- [Learning Human Pose Estimation Features with Convolutional Networks](https://mlsanity.com/paper/1312.7302/) (2013)

## Built on by (most cited first, 22 of 22)

- [Egocentric Whole-Body Motion Capture with FisheyeViT and Diffusion-Based Motion Refinement](https://mlsanity.com/paper/2311.16495/) (2023)
- [HMD^2: Environment-aware Motion Generation from Single Egocentric Head-Mounted Device](https://mlsanity.com/paper/2409.13426/) (2024)
- [3D Human Pose Perception from Egocentric Stereo Videos](https://mlsanity.com/paper/2401.00889/) (2023)
- [Challenges and Trends in Egocentric Vision: A Survey](https://mlsanity.com/paper/2503.15275/) (2025)
- [Attention-Propagation Network for Egocentric Heatmap to 3D Pose Lifting](https://mlsanity.com/paper/2402.18330/) (2024)
- [EMHI: A Multimodal Egocentric Human Motion Dataset with HMD and Body-Worn IMUs](https://mlsanity.com/paper/2408.17168/) (2024)
- [A Survey on 3D Egocentric Human Pose Estimation](https://mlsanity.com/paper/2403.17893/) (2024)
- [EgoAvatar: Egocentric View-Driven and Photorealistic Full-body Avatars](https://mlsanity.com/paper/2410.01835/) (2024)
- [Bring Your Rear Cameras for Egocentric 3D Human Pose Estimation](https://mlsanity.com/paper/2503.11652/) (2025)
- [EventEgo3D++: 3D Human Motion Capture from a Head-Mounted Event Camera](https://mlsanity.com/paper/2502.07869/) (2025)
- [FRAME: Floor-aligned Representation for Avatar Motion from Egocentric Video](https://mlsanity.com/paper/2503.23094/) (2025)
- [ECHO: Ego-Centric modeling of Human-Object interactions](https://mlsanity.com/paper/2508.21556/) (2025)
- [EgoPoseFormer: A Simple Baseline for Stereo Egocentric 3D Human Pose Estimation](https://mlsanity.com/paper/2403.18080/) (2024)
- [XR-MBT: Multi-modal Full Body Tracking for XR through Self-Supervision with Learned Depth Point Cloud Registration](https://mlsanity.com/paper/2411.18377/) (2024)
- [EgoPoseFormer v2: Accurate Egocentric Human Motion Estimation for AR/VR](https://mlsanity.com/paper/2603.04090/) (2026)
- [Event-based Egocentric Human Pose Estimation in Dynamic Environment](https://mlsanity.com/paper/2505.22007/) (2025)
- [E-3DPSM: A State Machine for Event-Based Egocentric 3D Human Pose Estimation](https://mlsanity.com/paper/2604.08543/) (2026)
- [Mean of Means: A 10-dollar Solution for Human Localization with Calibration-free and Unconstrained Camera Settings](https://mlsanity.com/paper/2407.20870/) (2024)
- [Mean of Means: Human Localization with Calibration-free and Unconstrained Camera Settings (extended version)](https://mlsanity.com/paper/2502.13017/) (2025)
- [EgoPoseVR: Spatiotemporal Multi-Modal Reasoning for Egocentric Full-Body Pose in Virtual Reality](https://mlsanity.com/paper/2602.05590/) (2026)
- [AG-EgoPose: Spatially Anchored Residual Correction with Action Context for Monocular Egocentric 3D Pose Estimation](https://mlsanity.com/paper/2603.25175/) (2026)
- [TSR-Ego: Temporally Guided Stereo Refinement Framework for Egocentric 3D Human Pose Estimation](https://mlsanity.com/paper/2607.09169/) (2026)

## Abstract (arXiv)

We present Ego3DPose, a highly accurate binocular egocentric 3D pose reconstruction system. The binocular egocentric setup offers practicality and usefulness in various applications, however, it remains largely under-explored. It has been suffering from low pose estimation accuracy due to viewing distortion, severe self-occlusion, and limited field-of-view of the joints in egocentric 2D images. Here, we notice that two important 3D cues, stereo correspondences, and perspective, contained in the egocentric binocular input are neglected. Current methods heavily rely on 2D image features, implicitly learning 3D information, which introduces biases towards commonly observed motions and leads to low overall accuracy. We observe that they not only fail in challenging occlusion cases but also in estimating visible joint positions. To address these challenges, we propose two novel approaches. First, we design a two-path network architecture with a path that estimates pose per limb independently with its binocular heatmaps. Without full-body information provided, it alleviates bias toward trained full-body distribution. Second, we leverage the egocentric view of body limbs, which exhibits strong perspective variance (e.g., a significantly large-size hand when it is close to the camera). We propose a new perspective-aware representation using trigonometry, enabling the network to estimate the 3D orientation of limbs. Finally, we develop an end-to-end pose reconstruction network that synergizes both techniques. Our comprehensive evaluations demonstrate that Ego3DPose outperforms state-of-the-art models by a pose estimation error (i.e., MPJPE) reduction of 23.1% in the UnrealEgo dataset. Our qualitative results highlight the superiority of our approach across a range of scenarios and challenges.