# EgoRenderer: Rendering Human Avatars from Egocentric Camera Images

arXiv: https://arxiv.org/abs/2111.12685 · HTML: https://mlsanity.com/paper/2111.12685/

## Key facts

- Cited by 13 papers in the map (data through 14 Oct 2026).
- Builds on 44 of its 61 references that are in the map.
- First posted on arXiv 24 Nov 2021.

## Builds on (most cited first, 30 of 44)

- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Very Deep Convolutional Networks for Large-Scale Image Recognition](https://mlsanity.com/paper/1409.1556/) (2014)
- [The Unreasonable Effectiveness of Deep Features as a Perceptual Metric](https://mlsanity.com/paper/1801.03924/) (2018)
- [Perceptual Losses for Real-Time Style Transfer and Super-Resolution](https://mlsanity.com/paper/1603.08155/) (2016)
- [High-Resolution Image Synthesis and Semantic Manipulation with Conditional GANs](https://mlsanity.com/paper/1711.11585/) (2017)
- [Neural Sparse Voxel Fields](https://mlsanity.com/paper/2007.11571/) (2020)
- [Scene Representation Networks: Continuous 3D-Structure-Aware Neural Scene Representations](https://mlsanity.com/paper/1906.01618/) (2019)
- [SphereFace: Deep Hypersphere Embedding for Face Recognition](https://mlsanity.com/paper/1704.08063/) (2017)
- [DensePose: Dense Human Pose Estimation In The Wild](https://mlsanity.com/paper/1802.00434/) (2018)
- [Monocular 3D Human Pose Estimation In The Wild Using Improved CNN Supervision](https://mlsanity.com/paper/1611.09813/) (2016)
- [PointRend: Image Segmentation as Rendering](https://mlsanity.com/paper/1912.08193/) (2019)
- [DeepVoxels: Learning Persistent 3D Feature Embeddings](https://mlsanity.com/paper/1812.01024/) (2018)
- [Video-to-Video Synthesis](https://mlsanity.com/paper/1808.06601/) (2018)
- [Learning from Synthetic Humans](https://mlsanity.com/paper/1701.01370/) (2017)
- [Everybody Dance Now](https://mlsanity.com/paper/1808.07371/) (2018)
- [Deep Video Portraits](https://mlsanity.com/paper/1805.11714/) (2018)
- [Deferred Neural Rendering: Image Synthesis using Neural Textures](https://mlsanity.com/paper/1904.12356/) (2019)
- [Pose Guided Person Image Generation](https://mlsanity.com/paper/1705.09368/) (2017)
- [Deformable GANs for Pose-based Human Image Generation](https://mlsanity.com/paper/1801.00055/) (2017)
- [Deep Appearance Models for Face Rendering](https://mlsanity.com/paper/1808.00362/) (2018)
- [Progressive Pose Attention Transfer for Person Image Generation](https://mlsanity.com/paper/1904.03349/) (2019)
- [VNect: Real-time 3D Human Pose Estimation with a Single RGB Camera](https://mlsanity.com/paper/1705.01583/) (2017)
- [Disentangled Person Image Generation](https://mlsanity.com/paper/1712.02621/) (2017)
- [Ego-Pose Estimation and Forecasting as Real-Time PD Control](https://mlsanity.com/paper/1906.03173/) (2019)
- [Real-time Hand Tracking under Occlusion from an Egocentric RGB-D Sensor](https://mlsanity.com/paper/1704.02201/) (2017)
- [Dense Pose Transfer](https://mlsanity.com/paper/1809.01995/) (2018)
- [xR-EgoPose: Egocentric 3D Human Pose from an HMD Camera](https://mlsanity.com/paper/1907.10045/) (2019)
- [Mo2Cap2: Real-time Mobile 3D Motion Capture with a Cap-mounted Fisheye Camera](https://mlsanity.com/paper/1803.05959/) (2018)
- [Textured Neural Avatars](https://mlsanity.com/paper/1905.08776/) (2019)
- [Neural Re-Rendering of Humans from a Single Image](https://mlsanity.com/paper/2101.04104/) (2021)

## Built on by (most cited first, 13 of 13)

- [HDR-GS: Efficient High Dynamic Range Novel View Synthesis at 1000x Speed via Gaussian Splatting](https://mlsanity.com/paper/2405.15125/) (2024)
- [HVTR: Hybrid Volumetric-Textural Rendering for Human Avatars](https://mlsanity.com/paper/2112.10203/) (2021)
- [HumanLiff: Layer-wise 3D Human Generation with Diffusion Model](https://mlsanity.com/paper/2308.09712/) (2023)
- [StructLDM: Structured Latent Diffusion for 3D Human Generation](https://mlsanity.com/paper/2404.01241/) (2024)
- [SurMo: Surface-based 4D Motion Modeling for Dynamic Human Rendering](https://mlsanity.com/paper/2404.01225/) (2024)
- [A Survey on 3D Egocentric Human Pose Estimation](https://mlsanity.com/paper/2403.17893/) (2024)
- [EgoAvatar: Egocentric View-Driven and Photorealistic Full-body Avatars](https://mlsanity.com/paper/2410.01835/) (2024)
- [Ground Reaction Inertial Poser: Physics-based Human Motion Capture from Sparse IMUs and Insole Pressure Sensors](https://mlsanity.com/paper/2603.16233/) (2026)
- [FashionEngine: Interactive 3D Human Generation and Editing via Multimodal Controls](https://mlsanity.com/paper/2404.01655/) (2024)
- [EgoAnimate: Generating Human Animations from Egocentric top-down Views](https://mlsanity.com/paper/2507.09230/) (2025)
- [InstantHDR: Single-forward Gaussian Splatting Initialization for HDR 3D Reconstruction](https://mlsanity.com/paper/2603.11298/) (2026)
- [HumANDiff: Articulated Noise Diffusion for Motion-Consistent Human Video Generation](https://mlsanity.com/paper/2604.05961/) (2026)
- [EgoRelight: Egocentric Human Capture and Illumination Recovery for Relightable and Photoreal Avatar Rendering](https://mlsanity.com/paper/2605.28401/) (2026)

## Abstract (arXiv)

We present EgoRenderer, a system for rendering full-body neural avatars of a person captured by a wearable, egocentric fisheye camera that is mounted on a cap or a VR headset. Our system renders photorealistic novel views of the actor and her motion from arbitrary virtual camera locations. Rendering full-body avatars from such egocentric images come with unique challenges due to the top-down view and large distortions. We tackle these challenges by decomposing the rendering process into several steps, including texture synthesis, pose construction, and neural image translation. For texture synthesis, we propose Ego-DPNet, a neural network that infers dense correspondences between the input fisheye images and an underlying parametric body model, and to extract textures from egocentric inputs. In addition, to encode dynamic appearances, our approach also learns an implicit texture stack that captures detailed appearance variation across poses and viewpoints. For correct pose generation, we first estimate body pose from the egocentric view using a parametric model. We then synthesize an external free-viewpoint pose image by projecting the parametric model to the user-specified target viewpoint. We next combine the target pose image and the textures into a combined feature image, which is transformed into the output color image using a neural image translation network. Experimental evaluations show that EgoRenderer is capable of generating realistic free-viewpoint avatars of a person wearing an egocentric camera. Comparisons to several baselines demonstrate the advantages of our approach.