# Fake It Till You Make It: Face analysis in the wild using synthetic data alone

arXiv: https://arxiv.org/abs/2109.15102 · HTML: https://mlsanity.com/paper/2109.15102/

## Key facts

- Cited by 200 papers in the map (data through 14 Oct 2026).
- Builds on 39 of its 81 references that are in the map.
- First posted on arXiv 30 Sep 2021.

## Builds on (most cited first, 30 of 39)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [PyTorch: An Imperative Style, High-Performance Deep Learning Library](https://mlsanity.com/paper/1912.01703/) (2019)
- [U-Net: Convolutional Networks for Biomedical Image Segmentation](https://mlsanity.com/paper/1505.04597/) (2015)
- [Domain-Adversarial Training of Neural Networks](https://mlsanity.com/paper/1505.07818/) (2015)
- [Playing for Data: Ground Truth from Computer Games](https://mlsanity.com/paper/1608.02192/) (2016)
- [PIFu: Pixel-Aligned Implicit Function for High-Resolution Clothed Human Digitization](https://mlsanity.com/paper/1905.05172/) (2019)
- [PIFuHD: Multi-Level Pixel-Aligned Implicit Function for High-Resolution 3D Human Digitization](https://mlsanity.com/paper/2004.00452/) (2020)
- [Learning from Synthetic Humans](https://mlsanity.com/paper/1701.01370/) (2017)
- [Virtual Worlds as Proxy for Multi-Object Tracking Analysis](https://mlsanity.com/paper/1605.06457/) (2016)
- [Learning from Simulated and Unsupervised Images through Adversarial Training](https://mlsanity.com/paper/1612.07828/) (2016)
- [Hand Keypoint Detection in Single Images using Multiview Bootstrapping](https://mlsanity.com/paper/1704.07809/) (2017)
- [Face Alignment in Full Pose Range: A 3D Total Solution](https://mlsanity.com/paper/1804.01005/) (2018)
- [GANerated Hands for Real-time 3D Hand Tracking from Monocular RGB](https://mlsanity.com/paper/1712.01057/) (2017)
- [Morphable Face Models - An Open Framework](https://mlsanity.com/paper/1709.08398/) (2017)
- [Look at Boundary: A Boundary-Aware Face Alignment Algorithm](https://mlsanity.com/paper/1805.10483/) (2018)
- [Adaptive Wing Loss for Robust Face Alignment via Heatmap Regression](https://mlsanity.com/paper/1904.07399/) (2019)
- [Meta-Sim: Learning to Generate Synthetic Datasets](https://mlsanity.com/paper/1904.11621/) (2019)
- [FairFace: Face Attribute Dataset for Balanced Race, Gender, and Age](https://mlsanity.com/paper/1908.04913/) (2019)
- [Learning Detailed Face Reconstruction from a Single Image](https://mlsanity.com/paper/1611.05053/) (2016)
- [Photorealistic Image Synthesis for Object Instance Detection](https://mlsanity.com/paper/1902.03334/) (2019)
- [3D Face Reconstruction by Learning from Synthetic Data](https://mlsanity.com/paper/1609.04387/) (2016)
- [CONFIG: Controllable Neural Face Image Generation](https://mlsanity.com/paper/2005.02671/) (2020)
- [Unrestricted Facial Geometry Reconstruction Using Image-to-Image Translation](https://mlsanity.com/paper/1703.10131/) (2017)
- [Rendering of Eyes for Eye-Shape Registration and Gaze Estimation](https://mlsanity.com/paper/1505.05916/) (2015)
- [Domain Adaptation through Synthesis for Unsupervised Person Re-identification](https://mlsanity.com/paper/1804.10094/) (2018)
- [Effective Use of Synthetic Data for Urban Scene Semantic Segmentation](https://mlsanity.com/paper/1807.06132/) (2018)
- [Simulating Content Consistent Vehicle Datasets with Attribute Descent](https://mlsanity.com/paper/1912.08855/) (2019)
- [Face Parsing with RoI Tanh-Warping](https://mlsanity.com/paper/1906.01342/) (2019)
- [3FabRec: Fast Few-shot Face alignment by Reconstruction](https://mlsanity.com/paper/1911.10448/) (2019)

## Built on by (most cited first, 30 of 200)

- [A Survey on Large Language Models for Code Generation](https://mlsanity.com/paper/2406.00515/) (2024)
- [Kubric: A scalable dataset generator](https://mlsanity.com/paper/2203.03570/) (2022)
- [Rodin: A Generative Model for Sculpting 3D Digital Avatars Using Diffusion](https://mlsanity.com/paper/2212.06135/) (2022)
- [What does CLIP know about a red circle? Visual prompt engineering for VLMs](https://mlsanity.com/paper/2304.06712/) (2023)
- [Infinite Photorealistic Worlds using Procedural Generation](https://mlsanity.com/paper/2306.09310/) (2023)
- [Machine Learning for Synthetic Data Generation: A Review](https://mlsanity.com/paper/2302.04062/) (2023)
- [CinePile: A Long Video Question Answering Dataset and Benchmark](https://mlsanity.com/paper/2405.08813/) (2024)
- [Best Practices and Lessons Learned on Synthetic Data](https://mlsanity.com/paper/2404.07503/) (2024)
- [DigiFace-1M: 1 Million Digital Face Images for Face Recognition](https://mlsanity.com/paper/2210.02579/) (2022)
- [BlockFusion: Expandable 3D Scene Generation using Latent Tri-plane Extrapolation](https://mlsanity.com/paper/2401.17053/) (2024)
- [IDiff-Face: Synthetic-based Face Recognition through Fizzy Identity-Conditioned Diffusion Models](https://mlsanity.com/paper/2308.04995/) (2023)
- [3D face reconstruction with dense landmarks](https://mlsanity.com/paper/2204.02776/) (2022)
- [MetaPortrait: Identity-Preserving Talking Head Generation with Fast Personalized Adaptation](https://mlsanity.com/paper/2212.08062/) (2022)
- [Learning to Relight Portrait Images via a Virtual Light Stage and Synthetic-to-Real Adaptation](https://mlsanity.com/paper/2209.10510/) (2022)
- [Real-Time Radiance Fields for Single-Image Portrait View Synthesis](https://mlsanity.com/paper/2305.02310/) (2023)
- [Portrait4D: Learning One-Shot 4D Head Avatar Synthesis using Synthetic Data](https://mlsanity.com/paper/2311.18729/) (2023)
- [4D-DRESS: A 4D Dataset of Real-world Human Clothing with Semantic Annotations](https://mlsanity.com/paper/2404.18630/) (2024)
- [On the Principles of Parsimony and Self-Consistency for the Emergence of Intelligence](https://mlsanity.com/paper/2207.04630/) (2022)
- [GaussianCube: A Structured and Explicit Radiance Representation for 3D Generative Modeling](https://mlsanity.com/paper/2403.19655/) (2024)
- [SimPer: Simple Self-Supervised Learning of Periodic Targets](https://mlsanity.com/paper/2210.03115/) (2022)
- [STAR Loss: Reducing Semantic Ambiguity in Facial Landmark Detection](https://mlsanity.com/paper/2306.02763/) (2023)
- [GAIA: Zero-shot Talking Avatar Generation](https://mlsanity.com/paper/2311.15230/) (2023)
- [HandRefiner: Refining Malformed Hands in Generated Images by Diffusion-based Conditional Inpainting](https://mlsanity.com/paper/2311.17957/) (2023)
- [3D Face Reconstruction with the Geometric Guidance of Facial Part Segmentation](https://mlsanity.com/paper/2312.00311/) (2023)
- [VoLux-GAN: A Generative Model for 3D Face Synthesis with HDRI Relighting](https://mlsanity.com/paper/2201.04873/) (2022)
- [AvatarBooth: High-Quality and Customizable 3D Human Avatar Generation](https://mlsanity.com/paper/2306.09864/) (2023)
- [VolTeMorph: Realtime, Controllable and Generalisable Animation of Volumetric Representations](https://mlsanity.com/paper/2208.00949/) (2022)
- [RodinHD: High-Fidelity 3D Avatar Generation with Diffusion Models](https://mlsanity.com/paper/2407.06938/) (2024)
- [ArtiFact: A Large-Scale Dataset with Artificial and Factual Images for Generalizable and Robust Synthetic Image Detection](https://mlsanity.com/paper/2302.11970/) (2023)
- [LRM-Zero: Training Large Reconstruction Models with Synthesized Data](https://mlsanity.com/paper/2406.09371/) (2024)

## Abstract (arXiv)

We demonstrate that it is possible to perform face-related computer vision in the wild using synthetic data alone. The community has long enjoyed the benefits of synthesizing training data with graphics, but the domain gap between real and synthetic data has remained a problem, especially for human faces. Researchers have tried to bridge this gap with data mixing, domain adaptation, and domain-adversarial training, but we show that it is possible to synthesize data with minimal domain gap, so that models trained on synthetic data generalize to real in-the-wild datasets. We describe how to combine a procedurally-generated parametric 3D face model with a comprehensive library of hand-crafted assets to render training images with unprecedented realism and diversity. We train machine learning systems for face-related tasks such as landmark localization and face parsing, showing that synthetic data can both match real data in accuracy as well as open up new approaches where manual labelling would be impossible.