# EgoAvatar: Egocentric View-Driven and Photorealistic Full-body Avatars arXiv: https://arxiv.org/abs/2410.01835 ยท HTML: https://mlsanity.com/paper/2410.01835/ ## Key facts - Cited by 8 papers in the map (data through 14 Oct 2026). - Builds on 44 of its 63 references that are in the map. - First posted on arXiv 22 Sep 2024. ## Builds on (most cited first, 30 of 44) - [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014) - [U-Net: Convolutional Networks for Biomedical Image Segmentation](https://mlsanity.com/paper/1505.04597/) (2015) - [The Unreasonable Effectiveness of Deep Features as a Perceptual Metric](https://mlsanity.com/paper/1801.03924/) (2018) - [Segment Anything](https://mlsanity.com/paper/2304.02643/) (2023) - [3D Gaussian Splatting for Real-Time Radiance Field Rendering](https://mlsanity.com/paper/2308.04079/) (2023) - [Accelerating 3D Deep Learning with PyTorch3D](https://mlsanity.com/paper/2007.08501/) (2020) - [Neural Body: Implicit Neural Representations with Structured Latent Codes for Novel View Synthesis of Dynamic Humans](https://mlsanity.com/paper/2012.15838/) (2020) - [HumanNeRF: Free-viewpoint Rendering of Moving People from Monocular Video](https://mlsanity.com/paper/2201.04127/) (2022) - [A simple yet effective baseline for 3d human pose estimation](https://mlsanity.com/paper/1705.03098/) (2017) - [NeuS2: Fast Learning of Neural Implicit Surfaces for Multi-view Reconstruction](https://mlsanity.com/paper/2212.05231/) (2022) - [Mixture of Volumetric Primitives for Efficient Neural Rendering](https://mlsanity.com/paper/2103.01954/) (2021) - [NeuMan: Neural Human Radiance Field from a Single Video](https://mlsanity.com/paper/2203.12575/) (2022) - [Deep Appearance Models for Face Rendering](https://mlsanity.com/paper/1808.00362/) (2018) - [Neural Human Performer: Learning Generalizable Radiance Fields for Human Performance Rendering](https://mlsanity.com/paper/2109.07448/) (2021) - [Real-time Deep Dynamic Characters](https://mlsanity.com/paper/2105.01794/) (2021) - [Neural Actor: Neural Free-view Synthesis of Human Actors with Pose Control](https://mlsanity.com/paper/2106.02019/) (2021) - [ASH: Animatable Gaussian Splats for Efficient and Photoreal Human Rendering](https://mlsanity.com/paper/2312.05941/) (2023) - [Dynamics-Regulated Kinematic Policy for Egocentric Pose Estimation](https://mlsanity.com/paper/2106.05969/) (2021) - [Ego-Pose Estimation and Forecasting as Real-Time PD Control](https://mlsanity.com/paper/1906.03173/) (2019) - [AvatarReX: Real-time Expressive Full-body Avatars](https://mlsanity.com/paper/2305.04789/) (2023) - [Image Inpainting via Generative Multi-column Convolutional Neural Networks](https://mlsanity.com/paper/1810.08771/) (2018) - [Driving-Signal Aware Full-Body Avatars](https://mlsanity.com/paper/2105.10441/) (2021) - [xR-EgoPose: Egocentric 3D Human Pose from an HMD Camera](https://mlsanity.com/paper/1907.10045/) (2019) - [Mo2Cap2: Real-time Mobile 3D Motion Capture with a Cap-mounted Fisheye Camera](https://mlsanity.com/paper/1803.05959/) (2018) - [Modeling Clothing as a Separate Layer for an Animatable Human Avatar](https://mlsanity.com/paper/2106.14879/) (2021) - [Drivable Volumetric Avatars using Texel-Aligned Features](https://mlsanity.com/paper/2207.09774/) (2022) - [UnrealEgo: A New Dataset for Robust Egocentric 3D Human Motion Capture](https://mlsanity.com/paper/2208.01633/) (2022) - [Estimating Egocentric 3D Human Pose in Global Space](https://mlsanity.com/paper/2104.13454/) (2021) - [DELIFFAS: Deformable Light Fields for Fast Avatar Synthesis](https://mlsanity.com/paper/2310.11449/) (2023) - [Estimating Egocentric 3D Human Pose in the Wild with External Weak Supervision](https://mlsanity.com/paper/2201.07929/) (2022) ## Built on by (most cited first, 8 of 8) - [RePerformer: Immersive Human-centric Volumetric Videos from Playback to Photoreal Reperformance](https://mlsanity.com/paper/2503.12242/) (2025) - [Real-time Free-view Human Rendering from Sparse-view RGB Videos using Double Unprojected Textures](https://mlsanity.com/paper/2412.13183/) (2024) - [FRAME: Floor-aligned Representation for Avatar Motion from Egocentric Video](https://mlsanity.com/paper/2503.23094/) (2025) - [Multimodal Knowledge Distillation for Egocentric Action Recognition Robust to Missing Modalities](https://mlsanity.com/paper/2504.08578/) (2025) - [EgoAnimate: Generating Human Animations from Egocentric top-down Views](https://mlsanity.com/paper/2507.09230/) (2025) - [EgoRelight: Egocentric Human Capture and Illumination Recovery for Relightable and Photoreal Avatar Rendering](https://mlsanity.com/paper/2605.28401/) (2026) - [Seeing the World and the Self from Egocentric Video](https://mlsanity.com/paper/2609.01276/) (2026) - [FlowAvatar: Real-Time Full-Body Avatars from Sparse Egocentric Inputs on Consumer XR Devices](https://mlsanity.com/paper/2610.16277/) (2026) ## Abstract (arXiv) Immersive VR telepresence ideally means being able to interact and communicate with digital avatars that are indistinguishable from and precisely reflect the behaviour of their real counterparts. The core technical challenge is two fold: Creating a digital double that faithfully reflects the real human and tracking the real human solely from egocentric sensing devices that are lightweight and have a low energy consumption, e.g. a single RGB camera. Up to date, no unified solution to this problem exists as recent works solely focus on egocentric motion capture, only model the head, or build avatars from multi-view captures. In this work, we, for the first time in literature, propose a person-specific egocentric telepresence approach, which jointly models the photoreal digital avatar while also driving it from a single egocentric video. We first present a character model that is animatible, i.e. can be solely driven by skeletal motion, while being capable of modeling geometry and appearance. Then, we introduce a personalized egocentric motion capture component, which recovers full-body motion from an egocentric video. Finally, we apply the recovered pose to our character model and perform a test-time mesh refinement such that the geometry faithfully projects onto the egocentric view. To validate our design choices, we propose a new and challenging benchmark, which provides paired egocentric and dense multi-view videos of real humans performing various motions. Our experiments demonstrate a clear step towards egocentric and photoreal telepresence as our method outperforms baselines as well as competing methods. For more details, code, and data, we refer to our project page.