# UNOC: Understanding Occlusion for Embodied Presence in Virtual Reality

arXiv: https://arxiv.org/abs/2012.03680 · HTML: https://mlsanity.com/paper/2012.03680/

## Key facts

- Cited by 7 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 12 Nov 2020.

## Built on by (most cited first, 7 of 7)

- [ARCH++: Animation-Ready Clothed Human Reconstruction Revisited](https://mlsanity.com/paper/2108.07845/) (2021)
- [EgoPoser: Robust Real-Time Egocentric Pose Estimation from Sparse and Intermittent Observations Everywhere](https://mlsanity.com/paper/2308.06493/) (2023)
- [PSVT: End-to-End Multi-person 3D Pose and Shape Estimation with Progressive Video Transformers](https://mlsanity.com/paper/2303.09187/) (2023)
- [Explicit Occlusion Reasoning for Multi-person 3D Human Pose Estimation](https://mlsanity.com/paper/2208.00090/) (2022)
- [Thermodynamics-informed neural networks for physically realistic mixed reality](https://mlsanity.com/paper/2210.13414/) (2022)
- [IVT: An End-to-End Instance-guided Video Transformer for 3D Pose Estimation](https://mlsanity.com/paper/2208.03431/) (2022)
- [FRESA: Feedforward Reconstruction of Personalized Skinned Avatars from Few Images](https://mlsanity.com/paper/2503.19207/) (2025)

## Abstract (arXiv)

Tracking body and hand motions in the 3D space is essential for social and self-presence in augmented and virtual environments. Unlike the popular 3D pose estimation setting, the problem is often formulated as inside-out tracking based on embodied perception (e.g., egocentric cameras, handheld sensors). In this paper, we propose a new data-driven framework for inside-out body tracking, targeting challenges of omnipresent occlusions in optimization-based methods (e.g., inverse kinematics solvers). We first collect a large-scale motion capture dataset with both body and finger motions using optical markers and inertial sensors. This dataset focuses on social scenarios and captures ground truth poses under self-occlusions and body-hand interactions. We then simulate the occlusion patterns in head-mounted camera views on the captured ground truth using a ray casting algorithm and learn a deep neural network to infer the occluded body parts. In the experiments, we show that our method is able to generate high-fidelity embodied poses by applying the proposed method on the task of real-time inside-out body tracking, finger motion synthesis, and 3-point inverse kinematics.