# 3D Human Pose Perception from Egocentric Stereo Videos

arXiv: https://arxiv.org/abs/2401.00889 · HTML: https://mlsanity.com/paper/2401.00889/

## Key facts

- Cited by 23 papers in the map (data through 14 Oct 2026).
- Builds on 35 of its 55 references that are in the map.
- First posted on arXiv 30 Dec 2023.

## Builds on (most cited first, 30 of 35)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Denoising Diffusion Probabilistic Models](https://mlsanity.com/paper/2006.11239/) (2020)
- [End-to-End Object Detection with Transformers](https://mlsanity.com/paper/2005.12872/) (2020)
- [Ego4D: Around the World in 3,000 Hours of Egocentric Video](https://mlsanity.com/paper/2110.07058/) (2021)
- [End-to-End Human Pose and Mesh Reconstruction with Transformers](https://mlsanity.com/paper/2012.09760/) (2020)
- [3D human pose estimation in video with temporal convolutions and semi-supervised training](https://mlsanity.com/paper/1811.11742/) (2018)
- [3D Human Pose Estimation with Spatial and Temporal Transformers](https://mlsanity.com/paper/2103.10455/) (2021)
- [MotionBERT: A Unified Perspective on Learning Human Motion Representations](https://mlsanity.com/paper/2210.06551/) (2022)
- [Aria Digital Twin: A New Benchmark Dataset for Egocentric 3D Machine Perception](https://mlsanity.com/paper/2306.06362/) (2023)
- [MixSTE: Seq2seq Mixed Spatio-Temporal Encoder for 3D Human Pose Estimation in Video](https://mlsanity.com/paper/2203.00859/) (2022)
- [MHFormer: Multi-Hypothesis Transformer for 3D Human Pose Estimation](https://mlsanity.com/paper/2111.12707/) (2021)
- [EgoBody: Human Body Shape and Motion of Interacting People from Head-Mounted Devices](https://mlsanity.com/paper/2112.07642/) (2021)
- [Epipolar Transformers](https://mlsanity.com/paper/2005.04551/) (2020)
- [Dynamics-Regulated Kinematic Policy for Egocentric Pose Estimation](https://mlsanity.com/paper/2106.05969/) (2021)
- [Exploiting Temporal Contexts with Strided Transformer for 3D Human Pose Estimation](https://mlsanity.com/paper/2103.14304/) (2021)
- [Ego-Body Pose Estimation via Ego-Head Pose Estimation](https://mlsanity.com/paper/2212.04636/) (2022)
- [PoseFormerV2: Exploring Frequency Domain for Efficient and Robust 3D Human Pose Estimation](https://mlsanity.com/paper/2303.17472/) (2023)
- [Ego-Pose Estimation and Forecasting as Real-Time PD Control](https://mlsanity.com/paper/1906.03173/) (2019)
- [xR-EgoPose: Egocentric 3D Human Pose from an HMD Camera](https://mlsanity.com/paper/1907.10045/) (2019)
- [Neural Monocular 3D Human Motion Capture with Physical Awareness](https://mlsanity.com/paper/2105.01057/) (2021)
- [Mo2Cap2: Real-time Mobile 3D Motion Capture with a Cap-mounted Fisheye Camera](https://mlsanity.com/paper/1803.05959/) (2018)
- [EgoHumans: An Egocentric 3D Multi-Human Benchmark](https://mlsanity.com/paper/2305.16487/) (2023)
- [UnrealEgo: A New Dataset for Robust Egocentric 3D Human Motion Capture](https://mlsanity.com/paper/2208.01633/) (2022)
- [Estimating Egocentric 3D Human Pose in Global Space](https://mlsanity.com/paper/2104.13454/) (2021)
- [SelfPose: 3D Egocentric Pose Estimation from a Headset Mounted Camera](https://mlsanity.com/paper/2011.01519/) (2020)
- [Estimating Egocentric 3D Human Pose in the Wild with External Weak Supervision](https://mlsanity.com/paper/2201.07929/) (2022)
- [Uplift and Upsample: Efficient 3D Human Pose Estimation with Uplifting Transformers](https://mlsanity.com/paper/2210.06110/) (2022)
- [Probabilistic Human Mesh Recovery in 3D Scenes from Egocentric Views](https://mlsanity.com/paper/2304.06024/) (2023)
- [TransFusion: Cross-view Fusion with Transformer for 3D Human Pose Estimation](https://mlsanity.com/paper/2110.09554/) (2021)

## Built on by (most cited first, 23 of 23)

- [EventEgo3D: 3D Human Motion Capture from Egocentric Event Streams](https://mlsanity.com/paper/2404.08640/) (2024)
- [HMD^2: Environment-aware Motion Generation from Single Egocentric Head-Mounted Device](https://mlsanity.com/paper/2409.13426/) (2024)
- [Challenges and Trends in Egocentric Vision: A Survey](https://mlsanity.com/paper/2503.15275/) (2025)
- [Ego4o: Egocentric Human Motion Capture and Understanding from Multi-Modal Input](https://mlsanity.com/paper/2504.08449/) (2025)
- [REWIND: Real-Time Egocentric Whole-Body Motion Diffusion with Exemplar-Based Identity Conditioning](https://mlsanity.com/paper/2504.04956/) (2025)
- [Intention-driven Ego-to-Exo Video Generation](https://mlsanity.com/paper/2403.09194/) (2024)
- [EMHI: A Multimodal Egocentric Human Motion Dataset with HMD and Body-Worn IMUs](https://mlsanity.com/paper/2408.17168/) (2024)
- [EgoCast: Forecasting Egocentric Human Pose in the Wild](https://mlsanity.com/paper/2412.02903/) (2024)
- [A Survey on 3D Egocentric Human Pose Estimation](https://mlsanity.com/paper/2403.17893/) (2024)
- [Bring Your Rear Cameras for Egocentric 3D Human Pose Estimation](https://mlsanity.com/paper/2503.11652/) (2025)
- [EventEgo3D++: 3D Human Motion Capture from a Head-Mounted Event Camera](https://mlsanity.com/paper/2502.07869/) (2025)
- [FRAME: Floor-aligned Representation for Avatar Motion from Egocentric Video](https://mlsanity.com/paper/2503.23094/) (2025)
- [Fish2Mesh Transformer: 3D Human Mesh Recovery from Egocentric Vision](https://mlsanity.com/paper/2503.06089/) (2025)
- [EgoCogNav: Cognition-aware Human Egocentric Navigation](https://mlsanity.com/paper/2511.17581/) (2025)
- [Stereo-Inertial Poser: Towards Metric-Accurate Shape-Aware Motion Capture Using Sparse IMUs and a Single Stereo Camera](https://mlsanity.com/paper/2603.02130/) (2026)
- [EgoPoseFormer v2: Accurate Egocentric Human Motion Estimation for AR/VR](https://mlsanity.com/paper/2603.04090/) (2026)
- [Event-based Egocentric Human Pose Estimation in Dynamic Environment](https://mlsanity.com/paper/2505.22007/) (2025)
- [Egocentric Visibility-Aware Human Pose Estimation](https://mlsanity.com/paper/2602.23618/) (2026)
- [E-3DPSM: A State Machine for Event-Based Egocentric 3D Human Pose Estimation](https://mlsanity.com/paper/2604.08543/) (2026)
- [EgoAnimate: Generating Human Animations from Egocentric top-down Views](https://mlsanity.com/paper/2507.09230/) (2025)
- [OmniEgoCap: Camera-Agnostic Sequence-Level Egocentric Motion Reconstruction](https://mlsanity.com/paper/2512.19283/) (2025)
- [Human Motion Estimation with Everyday Wearables](https://mlsanity.com/paper/2512.21209/) (2025)
- [TSR-Ego: Temporally Guided Stereo Refinement Framework for Egocentric 3D Human Pose Estimation](https://mlsanity.com/paper/2607.09169/) (2026)

## Abstract (arXiv)

While head-mounted devices are becoming more compact, they provide egocentric views with significant self-occlusions of the device user. Hence, existing methods often fail to accurately estimate complex 3D poses from egocentric views. In this work, we propose a new transformer-based framework to improve egocentric stereo 3D human pose estimation, which leverages the scene information and temporal context of egocentric stereo videos. Specifically, we utilize 1) depth features from our 3D scene reconstruction module with uniformly sampled windows of egocentric stereo frames, and 2) human joint queries enhanced by temporal features of the video inputs. Our method is able to accurately estimate human poses even in challenging scenarios, such as crouching and sitting. Furthermore, we introduce two new benchmark datasets, i.e., UnrealEgo2 and UnrealEgo-RW (RealWorld). The proposed datasets offer a much larger number of egocentric stereo views with a wider variety of human motions than the existing datasets, allowing comprehensive evaluation of existing and upcoming methods. Our extensive experiments show that the proposed approach significantly outperforms previous methods. We will release UnrealEgo2, UnrealEgo-RW, and trained models on our project page.