# MirrorNeRF: One-shot Neural Portrait Radiance Field from Multi-mirror Catadioptric Imaging

arXiv: https://arxiv.org/abs/2104.02607 · HTML: https://mlsanity.com/paper/2104.02607/

## Key facts

- Cited by 9 papers in the map (data through 7 Oct 2026).
- Builds on 19 of its 40 references that are in the map.
- First posted on arXiv 6 Apr 2021.

## Builds on (most cited first, 19 of 19)

- [The Unreasonable Effectiveness of Deep Features as a Perceptual Metric](https://mlsanity.com/paper/1801.03924/) (2018)
- [ShapeNet: An Information-Rich 3D Model Repository](https://mlsanity.com/paper/1512.03012/) (2015)
- [DeepSDF: Learning Continuous Signed Distance Functions for Shape Representation](https://mlsanity.com/paper/1901.05103/) (2019)
- [Occupancy Networks: Learning 3D Reconstruction in Function Space](https://mlsanity.com/paper/1812.03828/) (2018)
- [D-NeRF: Neural Radiance Fields for Dynamic Scenes](https://mlsanity.com/paper/2011.13961/) (2020)
- [NeRF in the Wild: Neural Radiance Fields for Unconstrained Photo Collections](https://mlsanity.com/paper/2008.02268/) (2020)
- [Learning Implicit Fields for Generative Shape Modeling](https://mlsanity.com/paper/1812.02822/) (2018)
- [Neural Sparse Voxel Fields](https://mlsanity.com/paper/2007.11571/) (2020)
- [Scene Representation Networks: Continuous 3D-Structure-Aware Neural Scene Representations](https://mlsanity.com/paper/1906.01618/) (2019)
- [NeRF++: Analyzing and Improving Neural Radiance Fields](https://mlsanity.com/paper/2010.07492/) (2020)
- [PIFu: Pixel-Aligned Implicit Function for High-Resolution Clothed Human Digitization](https://mlsanity.com/paper/1905.05172/) (2019)
- [Convolutional Occupancy Networks](https://mlsanity.com/paper/2003.04618/) (2020)
- [Differentiable Volumetric Rendering: Learning Implicit 3D Representations without 3D Supervision](https://mlsanity.com/paper/1912.07372/) (2019)
- [PIFuHD: Multi-Level Pixel-Aligned Implicit Function for High-Resolution 3D Human Digitization](https://mlsanity.com/paper/2004.00452/) (2020)
- [Space-time Neural Irradiance Fields for Free-Viewpoint Video](https://mlsanity.com/paper/2011.12950/) (2020)
- [Deferred Neural Rendering: Image Synthesis using Neural Textures](https://mlsanity.com/paper/1904.12356/) (2019)
- [Free View Synthesis](https://mlsanity.com/paper/2008.05511/) (2020)
- [A Neural Rendering Framework for Free-Viewpoint Relighting](https://mlsanity.com/paper/1911.11530/) (2019)
- [Deep Surface Light Fields](https://mlsanity.com/paper/1810.06514/) (2018)

## Built on by (most cited first, 9 of 9)

- [Neural Fields in Visual Computing and Beyond](https://mlsanity.com/paper/2111.11426/) (2021)
- [Neural Point Catacaustics for Novel-View Synthesis of Reflections](https://mlsanity.com/paper/2301.01087/) (2023)
- [Human Performance Modeling and Rendering via Neural Animated Mesh](https://mlsanity.com/paper/2209.08468/) (2022)
- [Neural Free-Viewpoint Performance Rendering under Complex Human-object Interactions](https://mlsanity.com/paper/2108.00362/) (2021)
- [Eikonal Fields for Refractive Novel-View Synthesis](https://mlsanity.com/paper/2202.00948/) (2022)
- [BeyondPixels: A Comprehensive Review of the Evolution of Neural Radiance Fields](https://mlsanity.com/paper/2306.03000/) (2023)
- [Dynamic Mesh-Aware Radiance Fields](https://mlsanity.com/paper/2309.04581/) (2023)
- [ORCa: Glossy Objects as Radiance Field Cameras](https://mlsanity.com/paper/2212.04531/) (2022)
- [VaxNeRF: Revisiting the Classic for Voxel-Accelerated Neural Radiance Field](https://mlsanity.com/paper/2111.13112/) (2021)

## Abstract (arXiv)

Photo-realistic neural reconstruction and rendering of the human portrait are critical for numerous VR/AR applications. Still, existing solutions inherently rely on multi-view capture settings, and the one-shot solution to get rid of the tedious multi-view synchronization and calibration remains extremely challenging. In this paper, we propose MirrorNeRF - a one-shot neural portrait free-viewpoint rendering approach using a catadioptric imaging system with multiple sphere mirrors and a single high-resolution digital camera, which is the first to combine neural radiance field with catadioptric imaging so as to enable one-shot photo-realistic human portrait reconstruction and rendering, in a low-cost and casual capture setting. More specifically, we propose a light-weight catadioptric system design with a sphere mirror array to enable diverse ray sampling in the continuous 3D space as well as an effective online calibration for the camera and the mirror array. Our catadioptric imaging system can be easily deployed with a low budget and the casual capture ability for convenient daily usages. We introduce a novel neural warping radiance field representation to learn a continuous displacement field that implicitly compensates for the misalignment due to our flexible system setting. We further propose a density regularization scheme to leverage the inherent geometry information from the catadioptric data in a self-supervision manner, which not only improves the training efficiency but also provides more effective density supervision for higher rendering quality. Extensive experiments demonstrate the effectiveness and robustness of our scheme to achieve one-shot photo-realistic and high-quality appearance free-viewpoint rendering for human portrait scenes.