# Seeing Invisible Poses: Estimating 3D Body Pose from Egocentric Video

arXiv: https://arxiv.org/abs/1603.07763 · HTML: https://mlsanity.com/paper/1603.07763/

## Key facts

- Cited by 45 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 24 Mar 2016.

## Built on by (most cited first, 30 of 45)

- [Ego4D: Around the World in 3,000 Hours of Egocentric Video](https://mlsanity.com/paper/2110.07058/) (2021)
- [Ego-Exo4D: Understanding Skilled Human Activity from First- and Third-Person Perspectives](https://mlsanity.com/paper/2311.18259/) (2023)
- [EgoBody: Human Body Shape and Motion of Interacting People from Head-Mounted Devices](https://mlsanity.com/paper/2112.07642/) (2021)
- [Human POSEitioning System (HPS): 3D Human Pose Estimation and Self-localization in Large Scenes from Body-Mounted Sensors](https://mlsanity.com/paper/2103.17265/) (2021)
- [Dynamics-Regulated Kinematic Policy for Egocentric Pose Estimation](https://mlsanity.com/paper/2106.05969/) (2021)
- [Ego-Exo: Transferring Visual Representations from Third-person to First-person Videos](https://mlsanity.com/paper/2104.07905/) (2021)
- [Ego-Body Pose Estimation via Ego-Head Pose Estimation](https://mlsanity.com/paper/2212.04636/) (2022)
- [An Outlook into the Future of Egocentric Vision](https://mlsanity.com/paper/2308.07123/) (2023)
- [GIMO: Gaze-Informed Human Motion Prediction in Context](https://mlsanity.com/paper/2204.09443/) (2022)
- [Estimating Egocentric 3D Human Pose in Global Space](https://mlsanity.com/paper/2104.13454/) (2021)
- [EgoLocate: Real-time Motion Capture, Localization, and Mapping with Sparse Body-mounted Sensors](https://mlsanity.com/paper/2305.01599/) (2023)
- [Retrieval-Augmented Egocentric Video Captioning](https://mlsanity.com/paper/2401.00789/) (2024)
- [Egocentric Audio-Visual Object Localization](https://mlsanity.com/paper/2303.13471/) (2023)
- [Probabilistic Human Mesh Recovery in 3D Scenes from Egocentric Views](https://mlsanity.com/paper/2304.06024/) (2023)
- [Egocentric Whole-Body Motion Capture with FisheyeViT and Diffusion-Based Motion Refinement](https://mlsanity.com/paper/2311.16495/) (2023)
- [Mocap Everyone Everywhere: Lightweight Motion Capture With Smartwatches and a Head-Mounted Camera](https://mlsanity.com/paper/2401.00847/) (2024)
- [Egocentric Pose Estimation from Human Vision Span](https://mlsanity.com/paper/2104.05167/) (2021)
- [Interaction Replica: Tracking Human-Object Interaction and Scene Changes From Human Motion](https://mlsanity.com/paper/2205.02830/) (2022)
- [EgoTracks: A Long-term Egocentric Visual Object Tracking Dataset](https://mlsanity.com/paper/2301.03213/) (2023)
- [HMD^2: Environment-aware Motion Generation from Single Egocentric Head-Mounted Device](https://mlsanity.com/paper/2409.13426/) (2024)
- [EgoLoc: Revisiting 3D Object Localization from Egocentric Videos with Visual Queries](https://mlsanity.com/paper/2212.06969/) (2022)
- [Shaping embodied agent behavior with activity-context priors from egocentric video](https://mlsanity.com/paper/2110.07692/) (2021)
- [Instance Tracking in 3D Scenes from Egocentric Videos](https://mlsanity.com/paper/2312.04117/) (2023)
- [Real-Time Simulated Avatar from Head-Mounted Sensors](https://mlsanity.com/paper/2403.06862/) (2024)
- [Ego4o: Egocentric Human Motion Capture and Understanding from Multi-Modal Input](https://mlsanity.com/paper/2504.08449/) (2025)
- [EgoRenderer: Rendering Human Avatars from Egocentric Camera Images](https://mlsanity.com/paper/2111.12685/) (2021)
- [EgoCast: Forecasting Egocentric Human Pose in the Wild](https://mlsanity.com/paper/2412.02903/) (2024)
- [EgoHDM: An Online Egocentric-Inertial Human Motion Capture, Localization, and Dense Mapping System](https://mlsanity.com/paper/2409.00343/) (2024)
- [HOOV: Hand Out-Of-View Tracking for Proprioceptive Interaction using Inertial Sensing](https://mlsanity.com/paper/2303.07016/) (2023)
- [A Survey on 3D Egocentric Human Pose Estimation](https://mlsanity.com/paper/2403.17893/) (2024)

## Abstract (arXiv)

Understanding the camera wearer's activity is central to egocentric vision, yet one key facet of that activity is inherently invisible to the camera--the wearer's body pose. Prior work focuses on estimating the pose of hands and arms when they come into view, but this 1) gives an incomplete view of the full body posture, and 2) prevents any pose estimate at all in many frames, since the hands are only visible in a fraction of daily life activities. We propose to infer the "invisible pose" of a person behind the egocentric camera. Given a single video, our efficient learning-based approach returns the full body 3D joint positions for each frame. Our method exploits cues from the dynamic motion signatures of the surrounding scene--which changes predictably as a function of body pose--as well as static scene structures that reveal the viewpoint (e.g., sitting vs. standing). We further introduce a novel energy minimization scheme to infer the pose sequence. It uses soft predictions of the poses per time instant together with a non-parametric model of human pose dynamics over longer windows. Our method outperforms an array of possible alternatives, including deep learning approaches for direct pose regression from images.