# LookinGoodπ: Real-time Person-independent Neural Re-rendering for High-quality Human Performance Capture

arXiv: https://arxiv.org/abs/2112.08037 · HTML: https://mlsanity.com/paper/2112.08037/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 24 of its 52 references that are in the map.
- First posted on arXiv 15 Dec 2021.

## Builds on (most cited first, 24 of 24)

- [The Unreasonable Effectiveness of Deep Features as a Perceptual Metric](https://mlsanity.com/paper/1801.03924/) (2018)
- [Expressive Body Capture: 3D Hands, Face, and Body from a Single Image](https://mlsanity.com/paper/1904.05866/) (2019)
- [Semantic Image Synthesis with Spatially-Adaptive Normalization](https://mlsanity.com/paper/1903.07291/) (2019)
- [OpenPose: Realtime Multi-Person 2D Pose Estimation using Part Affinity Fields](https://mlsanity.com/paper/1812.08008/) (2018)
- [First Order Motion Model for Image Animation](https://mlsanity.com/paper/2003.00196/) (2020)
- [One-Shot Free-View Neural Talking-Head Synthesis for Video Conferencing](https://mlsanity.com/paper/2011.15126/) (2020)
- [Neural Point-Based Graphics](https://mlsanity.com/paper/1906.08240/) (2019)
- [Total Capture: A 3D Deformation Model for Tracking Faces, Hands, and Bodies](https://mlsanity.com/paper/1801.01615/) (2018)
- [Animating Arbitrary Objects via Deep Motion Transfer](https://mlsanity.com/paper/1812.08861/) (2018)
- [Few-shot Video-to-Video Synthesis](https://mlsanity.com/paper/1910.12713/) (2019)
- [View Synthesis by Appearance Flow](https://mlsanity.com/paper/1605.03557/) (2016)
- [Neural Rerendering in the Wild](https://mlsanity.com/paper/1904.04290/) (2019)
- [Dense Intrinsic Appearance Flow for Human Pose Transfer](https://mlsanity.com/paper/1903.11326/) (2019)
- [Deep Image Spatial Transformation for Person Image Generation](https://mlsanity.com/paper/2003.00696/) (2020)
- [DoubleFusion: Real-time Capture of Human Performances with Inner Body Shapes from a Single Depth Sensor](https://mlsanity.com/paper/1804.06023/) (2018)
- [LiveCap: Real-time Human Performance Capture from Monocular Video](https://mlsanity.com/paper/1810.02648/) (2018)
- [Mask Guided Matting via Progressive Refinement Network](https://mlsanity.com/paper/2012.06722/) (2020)
- [Dense Pose Transfer](https://mlsanity.com/paper/1809.01995/) (2018)
- [SMPLpix: Neural Avatars from 3D Human Models](https://mlsanity.com/paper/2008.06872/) (2020)
- [StylePeople: A Generative Model of Fullbody Human Avatars](https://mlsanity.com/paper/2104.08363/) (2021)
- [NeuralHumanFVV: Real-Time Neural Volumetric Human Performance Rendering using RGB Cameras](https://mlsanity.com/paper/2103.07700/) (2021)
- [MUST-GAN: Multi-level Statistics Transfer for Self-driven Person Image Generation](https://mlsanity.com/paper/2011.09084/) (2020)
- [Volumetric Capture of Humans with a Single RGBD Camera via Semi-Parametric Learning](https://mlsanity.com/paper/1905.12162/) (2019)
- [Few-shot Neural Human Performance Rendering from Sparse RGBD Videos](https://mlsanity.com/paper/2107.06505/) (2021)

## Abstract (arXiv)

We propose LookinGoodπ, a novel neural re-rendering approach that is aimed to (1) improve the rendering quality of the low-quality reconstructed results from human performance capture system in real-time; (2) improve the generalization ability of the neural rendering network on unseen people. Our key idea is to utilize the rendered image of reconstructed geometry as the guidance to assist the prediction of person-specific details from few reference images, thus enhancing the re-rendered result. In light of this, we design a two-branch network. A coarse branch is designed to fix some artifacts (i.e. holes, noise) and obtain a coarse version of the rendered input, while a detail branch is designed to predict "correct" details from the warped references. The guidance of the rendered image is realized by blending features from two branches effectively in the training of the detail branch, which improves both the warping accuracy and the details' fidelity. We demonstrate that our method outperforms state-of-the-art methods at producing high-fidelity images on unseen people.