# MUNCH: Modelling Unique 'N Controllable Heads

arXiv: https://arxiv.org/abs/2310.02753 · HTML: https://mlsanity.com/paper/2310.02753/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 31 of its 49 references that are in the map.
- First posted on arXiv 4 Oct 2023.

## Builds on (most cited first, 30 of 31)

- [Image-to-Image Translation with Conditional Adversarial Networks](https://mlsanity.com/paper/1611.07004/) (2016)
- [Progressive Growing of GANs for Improved Quality, Stability, and Variation](https://mlsanity.com/paper/1710.10196/) (2017)
- [Training Generative Adversarial Networks with Limited Data](https://mlsanity.com/paper/2006.06676/) (2020)
- [Learning an Animatable Detailed 3D Face Model from In-The-Wild Images](https://mlsanity.com/paper/2012.04012/) (2020)
- [Generating 3D faces using Convolutional Mesh Autoencoders](https://mlsanity.com/paper/1807.10267/) (2018)
- [Capture, Learning, and Synthesis of 3D Speaking Styles](https://mlsanity.com/paper/1905.03079/) (2019)
- [Fake It Till You Make It: Face analysis in the wild using synthetic data alone](https://mlsanity.com/paper/2109.15102/) (2021)
- [GANFIT: Generative Adversarial Network Fitting for High Fidelity 3D Face Reconstruction](https://mlsanity.com/paper/1902.05978/) (2019)
- [Morphable Face Models - An Open Framework](https://mlsanity.com/paper/1709.08398/) (2017)
- [Learning Formation of Physically-Based Face Attributes](https://mlsanity.com/paper/2004.03458/) (2020)
- [AvatarMe: Realistically Renderable 3D Facial Reconstruction "in-the-wild"](https://mlsanity.com/paper/2003.13845/) (2020)
- [Neural 3D Morphable Models: Spiral Convolutional Networks for 3D Shape Representation Learning and Generation](https://mlsanity.com/paper/1905.02876/) (2019)
- [SpiralNet++: A Fast and Highly Efficient Mesh Convolution Operator](https://mlsanity.com/paper/1911.05856/) (2019)
- [Towards a complete 3D morphable model of the human head](https://mlsanity.com/paper/1911.08008/) (2019)
- [Unrestricted Facial Geometry Reconstruction Using Image-to-Image Translation](https://mlsanity.com/paper/1703.10131/) (2017)
- [Combining 3D Morphable Models: A Large scale Face-and-Head Model](https://mlsanity.com/paper/1903.03785/) (2019)
- [AvatarMe++: Facial Shape and BRDF Inference with Photorealistic Rendering-Aware GANs](https://mlsanity.com/paper/2112.05957/) (2021)
- [FitMe: Deep Photorealistic 3D Morphable Model Avatars](https://mlsanity.com/paper/2305.09641/) (2023)
- [OSTeC: One-Shot Texture Completion](https://mlsanity.com/paper/2012.15370/) (2020)
- [Synthesizing Coupled 3D Face Modalities by Trunk-Branch Generative Adversarial Networks](https://mlsanity.com/paper/1909.02215/) (2019)
- [Fast-GANFIT: Generative Adversarial Network for High Fidelity 3D Face Reconstruction](https://mlsanity.com/paper/2105.07474/) (2021)
- [Photorealistic Facial Texture Inference Using Deep Neural Networks](https://mlsanity.com/paper/1612.00523/) (2016)
- [DAD-3DHeads: A Large-scale Dense, Accurate and Diverse Dataset for 3D Head Alignment from a Single Image](https://mlsanity.com/paper/2204.03688/) (2022)
- [3DFaceGAN: Adversarial Nets for 3D Face Representation, Generation, and Translation](https://mlsanity.com/paper/1905.00307/) (2019)
- [High Quality Facial Surface and Texture Synthesis via Generative Adversarial Networks](https://mlsanity.com/paper/1808.08281/) (2018)
- [Dense 3D Face Decoding over 2500FPS: Joint Texture & Shape Convolutional Mesh Decoders](https://mlsanity.com/paper/1904.03525/) (2019)
- [A 3D GAN for Improved Large-pose Facial Recognition](https://mlsanity.com/paper/2012.10545/) (2020)
- [CA-GAN: Weakly Supervised Color Aware GAN for Controllable Makeup Transfer](https://mlsanity.com/paper/2008.10298/) (2020)
- [MeInGame: Create a Game Character Face from a Single Portrait](https://mlsanity.com/paper/2102.02371/) (2021)
- [3D Shape Variational Autoencoder Latent Disentanglement via Mini-Batch Feature Swapping for Bodies and Faces](https://mlsanity.com/paper/2111.12448/) (2021)

## Abstract (arXiv)

The automated generation of 3D human heads has been an intriguing and challenging task for computer vision researchers. Prevailing methods synthesize realistic avatars but with limited control over the diversity and quality of rendered outputs and suffer from limited correlation between shape and texture of the character. We propose a method that offers quality, diversity, control, and realism along with explainable network design, all desirable features to game-design artists in the domain. First, our proposed Geometry Generator identifies disentangled latent directions and generate novel and diverse samples. A Render Map Generator then learns to synthesize multiply high-fidelty physically-based render maps including Albedo, Glossiness, Specular, and Normals. For artists preferring fine-grained control over the output, we introduce a novel Color Transformer Model that allows semantic color control over generated maps. We also introduce quantifiable metrics called Uniqueness and Novelty and a combined metric to test the overall performance of our model. Demo for both shapes and textures can be found: https://munch-seven.vercel.app/. We will release our model along with the synthetic dataset.