# LookinGood: Enhancing Performance Capture with Real-time Neural Re-Rendering

arXiv: https://arxiv.org/abs/1811.05029 · HTML: https://mlsanity.com/paper/1811.05029/

## Key facts

- Cited by 22 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 12 Nov 2018.

## Built on by (most cited first, 22 of 22)

- [Advances in Neural Rendering](https://mlsanity.com/paper/2111.05849/) (2021)
- [MobileNeRF: Exploiting the Polygon Rasterization Pipeline for Efficient Neural Field Rendering on Mobile Architectures](https://mlsanity.com/paper/2208.00277/) (2022)
- [HyperNeRF: A Higher-Dimensional Representation for Topologically Varying Neural Radiance Fields](https://mlsanity.com/paper/2106.13228/) (2021)
- [KeypointNeRF: Generalizing Image-based Volumetric Avatars using Relative Spatial Encoding of Keypoints](https://mlsanity.com/paper/2205.04992/) (2022)
- [Neural Monocular 3D Human Motion Capture with Physical Awareness](https://mlsanity.com/paper/2105.01057/) (2021)
- [Neural Re-Rendering of Humans from a Single Image](https://mlsanity.com/paper/2101.04104/) (2021)
- [Self-supervised Outdoor Scene Relighting](https://mlsanity.com/paper/2107.03106/) (2021)
- [RobustFusion: Robust Volumetric Performance Reconstruction under Human-object Interactions from Monocular RGBD Stream](https://mlsanity.com/paper/2104.14837/) (2021)
- [Drivable Avatar Clothing: Faithful Full-Body Telepresence with Dynamic Clothing Driven by Sparse RGB-D Input](https://mlsanity.com/paper/2310.05917/) (2023)
- [FlexNeRF: Photorealistic Free-viewpoint Rendering of Moving Humans from Sparse Views](https://mlsanity.com/paper/2303.14368/) (2023)
- [PersonNeRF: Personalized Reconstruction from Photo Collections](https://mlsanity.com/paper/2302.08504/) (2023)
- [Tele-Aloha: A Low-budget and High-authenticity Telepresence System Using Sparse RGB Cameras](https://mlsanity.com/paper/2405.14866/) (2024)
- [Thermodynamics-informed neural networks for physically realistic mixed reality](https://mlsanity.com/paper/2210.13414/) (2022)
- [Artemis: Articulated Neural Pets with Appearance and Motion synthesis](https://mlsanity.com/paper/2202.05628/) (2022)
- [NeuralPassthrough: Learned Real-Time View Synthesis for VR](https://mlsanity.com/paper/2207.02186/) (2022)
- [EgoAvatar: Egocentric View-Driven and Photorealistic Full-body Avatars](https://mlsanity.com/paper/2410.01835/) (2024)
- [Deep Portrait Delighting](https://mlsanity.com/paper/2203.12088/) (2022)
- [Single-View View Synthesis with Self-Rectified Pseudo-Stereo](https://mlsanity.com/paper/2304.09527/) (2023)
- [MoCo-Flow: Neural Motion Consensus Flow for Dynamic Humans in Stationary Monocular Cameras](https://mlsanity.com/paper/2106.04477/) (2021)
- [Unsupervised Learning of Style-Aware Facial Animation from Real Acting Performances](https://mlsanity.com/paper/2306.10006/) (2023)
- [Uni-Render: A Unified Accelerator for Real-Time Rendering Across Diverse Neural Renderers](https://mlsanity.com/paper/2503.23644/) (2025)
- [Shape and Reflectance Reconstruction in Uncontrolled Environments by Differentiable Rendering](https://mlsanity.com/paper/2110.12975/) (2021)

## Abstract (arXiv)

Motivated by augmented and virtual reality applications such as telepresence, there has been a recent focus in real-time performance capture of humans under motion. However, given the real-time constraint, these systems often suffer from artifacts in geometry and texture such as holes and noise in the final rendering, poor lighting, and low-resolution textures. We take the novel approach to augment such real-time performance capture systems with a deep architecture that takes a rendering from an arbitrary viewpoint, and jointly performs completion, super resolution, and denoising of the imagery in real-time. We call this approach neural (re-)rendering, and our live system "LookinGood". Our deep architecture is trained to produce high resolution and high quality images from a coarse rendering in real-time. First, we propose a self-supervised training method that does not require manual ground-truth annotation. We contribute a specialized reconstruction error that uses semantic information to focus on relevant parts of the subject, e.g. the face. We also introduce a salient reweighing scheme of the loss function that is able to discard outliers. We specifically design the system for virtual and augmented reality headsets where the consistency between the left and right eye plays a crucial role in the final user experience. Finally, we generate temporally stable results by explicitly minimizing the difference between two consecutive frames. We tested the proposed system in two different scenarios: one involving a single RGB-D sensor, and upper body reconstruction of an actor, the second consisting of full body 360 degree capture. Through extensive experimentation, we demonstrate how our system generalizes across unseen sequences and subjects. The supplementary video is available at http://youtu.be/Md3tdAKoLGU.