# PersonNeRF: Personalized Reconstruction from Photo Collections

arXiv: https://arxiv.org/abs/2302.08504 · HTML: https://mlsanity.com/paper/2302.08504/

## Key facts

- Cited by 12 papers in the map (data through 14 Oct 2026).
- Builds on 28 of its 57 references that are in the map.
- First posted on arXiv 16 Feb 2023.

## Builds on (most cited first, 28 of 28)

- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [The Unreasonable Effectiveness of Deep Features as a Perceptual Metric](https://mlsanity.com/paper/1801.03924/) (2018)
- [NeRF in the Wild: Neural Radiance Fields for Unconstrained Photo Collections](https://mlsanity.com/paper/2008.02268/) (2020)
- [Block-NeRF: Scalable Large Scene Neural View Synthesis](https://mlsanity.com/paper/2202.05263/) (2022)
- [Neural Body: Implicit Neural Representations with Structured Latent Codes for Novel View Synthesis of Dynamic Humans](https://mlsanity.com/paper/2012.15838/) (2020)
- [Learning to Reconstruct 3D Human Pose and Shape via Model-fitting in the Loop](https://mlsanity.com/paper/1909.12828/) (2019)
- [RegNeRF: Regularizing Neural Radiance Fields for View Synthesis from Sparse Inputs](https://mlsanity.com/paper/2112.00724/) (2021)
- [Neural Fields in Visual Computing and Beyond](https://mlsanity.com/paper/2111.11426/) (2021)
- [HumanNeRF: Free-viewpoint Rendering of Moving People from Monocular Video](https://mlsanity.com/paper/2201.04127/) (2022)
- [Advances in Neural Rendering](https://mlsanity.com/paper/2111.05849/) (2021)
- [State of the Art on Neural Rendering](https://mlsanity.com/paper/2004.03805/) (2020)
- [NeuMan: Neural Human Radiance Field from a Single Video](https://mlsanity.com/paper/2203.12575/) (2022)
- [SNARF: Differentiable Forward Skinning for Animating Non-Rigid Neural Implicit Shapes](https://mlsanity.com/paper/2104.03953/) (2021)
- [SCANimate: Weakly Supervised Learning of Skinned Clothed Avatar Networks](https://mlsanity.com/paper/2104.03313/) (2021)
- [Neural Volumes: Learning Dynamic Renderable Volumes from Images](https://mlsanity.com/paper/1906.07751/) (2019)
- [TAVA: Template-free Animatable Volumetric Actors](https://mlsanity.com/paper/2206.08929/) (2022)
- [Optimizing the Latent Space of Generative Networks](https://mlsanity.com/paper/1707.05776/) (2017)
- [Neural Rerendering in the Wild](https://mlsanity.com/paper/1904.04290/) (2019)
- [LEAP: Learning Articulated Occupancy of People](https://mlsanity.com/paper/2104.06849/) (2021)
- [Neural Actor: Neural Free-view Synthesis of Human Actors with Pose Control](https://mlsanity.com/paper/2106.02019/) (2021)
- [Neural-GIF: Neural Generalized Implicit Functions for Animating People in Clothing](https://mlsanity.com/paper/2108.08807/) (2021)
- [Neural 3D Reconstruction in the Wild](https://mlsanity.com/paper/2205.12955/) (2022)
- [SCALE: Modeling Clothed Humans with a Surface Codec of Articulated Local Elements](https://mlsanity.com/paper/2104.07660/) (2021)
- [Crowdsampling the Plenoptic Function](https://mlsanity.com/paper/2007.15194/) (2020)
- [PINA: Learning a Personalized Implicit Neural Avatar from a Single RGB-D Video Sequence](https://mlsanity.com/paper/2203.01754/) (2022)
- [LookinGood: Enhancing Performance Capture with Real-time Neural Re-Rendering](https://mlsanity.com/paper/1811.05029/) (2018)
- [3D Time-lapse Reconstruction from Internet Photos](https://mlsanity.com/paper/1511.03019/) (2015)
- [Head Reconstruction from Internet Photos](https://mlsanity.com/paper/1809.04763/) (2018)

## Built on by (most cited first, 12 of 12)

- [LN3DIFF++: Scalable Latent Neural Fields Diffusion for Speedy 3D Generation](https://mlsanity.com/paper/2403.12019/) (2024)
- [Rendering Humans from Object-Occluded Monocular Videos](https://mlsanity.com/paper/2308.04622/) (2023)
- [A Survey on 3D Human Avatar Modeling -- From Reconstruction to Generation](https://mlsanity.com/paper/2406.04253/) (2024)
- [DreamWaltz-G: Expressive 3D Gaussian Avatars from Skeleton-Guided 2D Diffusion](https://mlsanity.com/paper/2409.17145/) (2024)
- [Generalizable Neural Voxels for Fast Human Radiance Fields](https://mlsanity.com/paper/2303.15387/) (2023)
- [PICA: Physics-Integrated Clothed Avatar](https://mlsanity.com/paper/2407.05324/) (2024)
- [IntrinsicNGP: Intrinsic Coordinate based Hash Encoding for Human NeRF](https://mlsanity.com/paper/2302.14683/) (2023)
- [Motion-Oriented Compositional Neural Radiance Fields for Monocular Dynamic Human Modeling](https://mlsanity.com/paper/2407.11962/) (2024)
- [AniDress: Animatable Loose-Dressed Avatar from Sparse Views Using Garment Rigging Model](https://mlsanity.com/paper/2401.15348/) (2024)
- [PAV: Personalized Head Avatar from Unstructured Video Collection](https://mlsanity.com/paper/2407.21047/) (2024)
- [Semantic-Human: Neural Rendering of Humans from Monocular Video with Human Parsing](https://mlsanity.com/paper/2308.09894/) (2023)
- [TimeWalker: Personalized Neural Space for Lifelong Head Avatars](https://mlsanity.com/paper/2412.02421/) (2024)

## Abstract (arXiv)

We present PersonNeRF, a method that takes a collection of photos of a subject (e.g. Roger Federer) captured across multiple years with arbitrary body poses and appearances, and enables rendering the subject with arbitrary novel combinations of viewpoint, body pose, and appearance. PersonNeRF builds a customized neural volumetric 3D model of the subject that is able to render an entire space spanned by camera viewpoint, body pose, and appearance. A central challenge in this task is dealing with sparse observations; a given body pose is likely only observed by a single viewpoint with a single appearance, and a given appearance is only observed under a handful of different body poses. We address this issue by recovering a canonical T-pose neural volumetric representation of the subject that allows for changing appearance across different observations, but uses a shared pose-dependent motion field across all observations. We demonstrate that this approach, along with regularization of the recovered volumetric geometry to encourage smoothness, is able to recover a model that renders compelling images from novel combinations of viewpoint, pose, and appearance from these challenging unstructured photo collections, outperforming prior work for free-viewpoint human rendering.