# Multiface: A Dataset for Neural Face Rendering

arXiv: https://arxiv.org/abs/2207.11243 · HTML: https://mlsanity.com/paper/2207.11243/

## Key facts

- Cited by 87 papers in the map (data through 14 Oct 2026).
- Builds on 13 of its 23 references that are in the map.
- First posted on arXiv 22 Jul 2022.

## Builds on (most cited first, 13 of 13)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [D-NeRF: Neural Radiance Fields for Dynamic Scenes](https://mlsanity.com/paper/2011.13961/) (2020)
- [Modular Primitives for High-Performance Differentiable Rendering](https://mlsanity.com/paper/2011.03277/) (2020)
- [Mixture of Volumetric Primitives for Efficient Neural Rendering](https://mlsanity.com/paper/2103.01954/) (2021)
- [MeshTalk: 3D Face Animation from Speech using Cross-Modality Disentanglement](https://mlsanity.com/paper/2104.08223/) (2021)
- [Neural Volumes: Learning Dynamic Renderable Volumes from Images](https://mlsanity.com/paper/1906.07751/) (2019)
- [Deep Appearance Models for Face Rendering](https://mlsanity.com/paper/1808.00362/) (2018)
- [Pixel Codec Avatars](https://mlsanity.com/paper/2104.04638/) (2021)
- [Learning Compositional Radiance Fields of Dynamic Human Heads](https://mlsanity.com/paper/2012.09955/) (2020)
- [Deforming Autoencoders: Unsupervised Disentangling of Shape and Appearance](https://mlsanity.com/paper/1806.06503/) (2018)
- [Audio- and Gaze-driven Facial Animation of Codec Avatars](https://mlsanity.com/paper/2008.05023/) (2020)
- [Expressive Telepresence via Modular Codec Avatars](https://mlsanity.com/paper/2008.11789/) (2020)

## Built on by (most cited first, 30 of 87)

- [NeRSemble: Multi-view Radiance Field Reconstruction of Human Heads](https://mlsanity.com/paper/2305.03027/) (2023)
- [Generating Holistic 3D Human Motion from Speech](https://mlsanity.com/paper/2212.04420/) (2022)
- [DNA-Rendering: A Diverse Neural Actor Repository for High-Fidelity Human-centric Rendering](https://mlsanity.com/paper/2307.10173/) (2023)
- [Emotional Speech-Driven Animation with Content-Emotion Disentanglement](https://mlsanity.com/paper/2306.08990/) (2023)
- [SMIRK: 3D Facial Expressions through Analysis-by-Neural-Synthesis](https://mlsanity.com/paper/2404.04104/) (2024)
- [FaceDiffuser: Speech-Driven 3D Facial Animation Synthesis Using Diffusion](https://mlsanity.com/paper/2309.11306/) (2023)
- [DiffusionRig: Learning Personalized Priors for Facial Appearance Editing](https://mlsanity.com/paper/2304.06711/) (2023)
- [PermutoSDF: Fast Multi-View Reconstruction with Implicit Surfaces using Permutohedral Lattices](https://mlsanity.com/paper/2211.12562/) (2022)
- [OTAvatar: One-shot Talking Face Avatar with Controllable Tri-plane Rendering](https://mlsanity.com/paper/2303.14662/) (2023)
- [NPGA: Neural Parametric Gaussian Avatars](https://mlsanity.com/paper/2405.19331/) (2024)
- [RenderMe-360: A Large Digital Asset Library and Benchmarks Towards High-fidelity Head Avatars](https://mlsanity.com/paper/2305.13353/) (2023)
- [DiffusionAvatars: Deferred Diffusion for High-fidelity 3D Head Avatars](https://mlsanity.com/paper/2311.18635/) (2023)
- [Media2Face: Co-speech Facial Animation Generation With Multi-Modality Guidance](https://mlsanity.com/paper/2401.15687/) (2024)
- [Towards Practical Capture of High-Fidelity Relightable Avatars](https://mlsanity.com/paper/2309.04247/) (2023)
- [State of the Art in Dense Monocular Non-Rigid 3D Reconstruction](https://mlsanity.com/paper/2210.15664/) (2022)
- [UniTalker: Scaling up Audio-Driven 3D Facial Animation through A Unified Model](https://mlsanity.com/paper/2408.00762/) (2024)
- [FaceXHuBERT: Text-less Speech-driven E(X)pressive 3D Facial Animation Synthesis Using Self-Supervised Speech Representation Learning](https://mlsanity.com/paper/2303.05416/) (2023)
- [LaughTalk: Expressive 3D Talking Head Generation with Laughter](https://mlsanity.com/paper/2311.00994/) (2023)
- [Neural Face Rigging for Animating and Retargeting Facial Meshes in the Wild](https://mlsanity.com/paper/2305.08296/) (2023)
- [CVTHead: One-shot Controllable Head Avatar with Vertex-feature Transformer](https://mlsanity.com/paper/2311.06443/) (2023)
- [ScanTalk: 3D Talking Heads from Unregistered Scans](https://mlsanity.com/paper/2403.10942/) (2024)
- [Morphable Diffusion: 3D-Consistent Diffusion for Single-image Avatar Creation](https://mlsanity.com/paper/2401.04728/) (2024)
- [LPFF: A Portrait Dataset for Face Generators Across Large Poses](https://mlsanity.com/paper/2303.14407/) (2023)
- [Gaussian3Diff: 3D Gaussian Diffusion for 3D Full Head Synthesis and Editing](https://mlsanity.com/paper/2312.03763/) (2023)
- [Gaussian Eigen Models for Human Heads](https://mlsanity.com/paper/2407.04545/) (2024)
- [3D Face Tracking from 2D Video through Iterative Dense UV to Image Flow](https://mlsanity.com/paper/2404.09819/) (2024)
- [DINER: Depth-aware Image-based NEural Radiance fields](https://mlsanity.com/paper/2211.16630/) (2022)
- [VRMM: A Volumetric Relightable Morphable Head Model](https://mlsanity.com/paper/2402.04101/) (2024)
- [Joker: Conditional 3D Head Synthesis with Extreme Facial Expressions](https://mlsanity.com/paper/2410.16395/) (2024)
- [GANHead: Towards Generative Animatable Neural Head Avatars](https://mlsanity.com/paper/2304.03950/) (2023)

## Abstract (arXiv)

Photorealistic avatars of human faces have come a long way in recent years, yet research along this area is limited by a lack of publicly available, high-quality datasets covering both, dense multi-view camera captures, and rich facial expressions of the captured subjects. In this work, we present Multiface, a new multi-view, high-resolution human face dataset collected from 13 identities at Reality Labs Research for neural face rendering. We introduce Mugsy, a large scale multi-camera apparatus to capture high-resolution synchronized videos of a facial performance. The goal of Multiface is to close the gap in accessibility to high quality data in the academic community and to enable research in VR telepresence. Along with the release of the dataset, we conduct ablation studies on the influence of different model architectures toward the model's interpolation capacity of novel viewpoint and expressions. With a conditional VAE model serving as our baseline, we found that adding spatial bias, texture warp field, and residual connections improves performance on novel view synthesis. Our code and data is available at: https://github.com/facebookresearch/multiface