# GANDiffFace: Controllable Generation of Synthetic Datasets for Face Recognition with Realistic Variations

arXiv: https://arxiv.org/abs/2305.19962 · HTML: https://mlsanity.com/paper/2305.19962/

## Key facts

- Cited by 49 papers in the map (data through 14 Oct 2026).
- Placed in Image generation, text-to-image and editing, one of 14 directions in Diffusion models.
- Builds on 30 of its 47 references that are in the map.
- First posted on arXiv 31 May 2023.

## Placed in

- Diffusion models / [Image generation, text-to-image and editing](https://mlsanity.com/diffusion/image-generation-text-to-image-and-editing/)

## Builds on (most cited first, 30 of 30)

- [Denoising Diffusion Probabilistic Models](https://mlsanity.com/paper/2006.11239/) (2020)
- [High-Resolution Image Synthesis with Latent Diffusion Models](https://mlsanity.com/paper/2112.10752/) (2021)
- [Diffusion Models Beat GANs on Image Synthesis](https://mlsanity.com/paper/2105.05233/) (2021)
- [Hierarchical Text-Conditional Image Generation with CLIP Latents](https://mlsanity.com/paper/2204.06125/) (2022)
- [Photorealistic Text-to-Image Diffusion Models with Deep Language Understanding](https://mlsanity.com/paper/2205.11487/) (2022)
- [A Style-Based Generator Architecture for Generative Adversarial Networks](https://mlsanity.com/paper/1812.04948/) (2018)
- [DreamBooth: Fine Tuning Text-to-Image Diffusion Models for Subject-Driven Generation](https://mlsanity.com/paper/2208.12242/) (2022)
- [Alias-Free Generative Adversarial Networks](https://mlsanity.com/paper/2106.12423/) (2021)
- [Variational Diffusion Models](https://mlsanity.com/paper/2107.00630/) (2021)
- [VGGFace2: A dataset for recognising faces across pose and age](https://mlsanity.com/paper/1710.08092/) (2017)
- [Interpreting the Latent Space of GANs for Semantic Face Editing](https://mlsanity.com/paper/1907.10786/) (2019)
- [MagFace: A Universal Representation for Face Recognition and Quality Assessment](https://mlsanity.com/paper/2103.06627/) (2021)
- [Towards Fast, Accurate and Stable 3D Dense Face Alignment](https://mlsanity.com/paper/2009.09960/) (2020)
- [DCFace: Synthetic Face Generation with Dual Condition Diffusion Model](https://mlsanity.com/paper/2304.07060/) (2023)
- [SynFace: Face Recognition with Synthetic Data](https://mlsanity.com/paper/2108.07960/) (2021)
- [DigiFace-1M: 1 Million Digital Face Images for Face Recognition](https://mlsanity.com/paper/2210.02579/) (2022)
- [Dive into Ambiguity: Latent Distribution Mining and Pairwise Uncertainty Estimation for Facial Expression Recognition](https://mlsanity.com/paper/2104.00232/) (2021)
- [Sample and Computation Redistribution for Efficient Face Detection](https://mlsanity.com/paper/2105.04714/) (2021)
- [SFace: Privacy-friendly and Accurate Face Recognition using Synthetic Data](https://mlsanity.com/paper/2206.10520/) (2022)
- [Third Time's the Charm? Image and Video Editing with StyleGAN3](https://mlsanity.com/paper/2201.13433/) (2022)
- [GANprintR: Improved Fakes and Evaluation of the State of the Art in Face Manipulation Detection](https://mlsanity.com/paper/1911.05351/) (2019)
- [Synthetic Data for Face Recognition: Current State and Future Prospects](https://mlsanity.com/paper/2305.01021/) (2023)
- [Studying Bias in GANs through the Lens of Race](https://mlsanity.com/paper/2209.02836/) (2022)
- [Synthetic Data in Human Analysis: A Survey](https://mlsanity.com/paper/2208.09191/) (2022)
- [On the use of automatically generated synthetic image datasets for benchmarking face recognition](https://mlsanity.com/paper/2106.04215/) (2021)
- [A Prompt Log Analysis of Text-to-Image Generation Systems](https://mlsanity.com/paper/2303.04587/) (2023)
- [Training Deep Face Recognition Systems with Synthetic Data](https://mlsanity.com/paper/1802.05891/) (2018)
- [On the Applicability of Synthetic Data for Face Recognition](https://mlsanity.com/paper/2104.02815/) (2021)
- [Child Face Recognition at Scale: Synthetic Data Generation and Performance Benchmark](https://mlsanity.com/paper/2304.11685/) (2023)
- [Attacking Face Recognition with T-shirts: Database, Vulnerability Assessment and Detection](https://mlsanity.com/paper/2211.07383/) (2022)

## Built on by (most cited first, 30 of 49)

- [FFAA: Multimodal Large Language Model based Explainable Open-World Face Forgery Analysis Assistant](https://mlsanity.com/paper/2408.10072/) (2024)
- [Balancing Act: Distribution-Guided Debiasing in Diffusion Models](https://mlsanity.com/paper/2402.18206/) (2024)
- [SDFR: Synthetic Data for Face Recognition Competition](https://mlsanity.com/paper/2404.04580/) (2024)
- [FRCSyn Challenge at WACV 2024:Face Recognition Challenge in the Era of Synthetic Data](https://mlsanity.com/paper/2311.10476/) (2023)
- [Toward Generalizable Forgery Detection and Reasoning](https://mlsanity.com/paper/2503.21210/) (2025)
- [Is My Data in Your AI? Membership Inference Test (MINT) applied to Face Biometrics](https://mlsanity.com/paper/2402.09225/) (2024)
- [Second FRCSyn-onGoing: Winning Solutions and Post-Challenge Analysis to Improve Face Recognition with Synthetic Data](https://mlsanity.com/paper/2412.01383/) (2024)
- [MIRROR: Multimodal Cognitive Reframing Therapy for Rolling with Resistance](https://mlsanity.com/paper/2504.13211/) (2025)
- [Synthetic Face Datasets Generation via Latent Space Exploration from Brownian Identity Diffusion](https://mlsanity.com/paper/2405.00228/) (2024)
- [HyperFace: Generating Synthetic Face Recognition Datasets by Exploring Face Embedding Hypersphere](https://mlsanity.com/paper/2411.08470/) (2024)
- [Synthetic Data for the Mitigation of Demographic Biases in Face Recognition](https://mlsanity.com/paper/2402.01472/) (2024)
- [ONOT: a High-Quality ICAO-compliant Synthetic Mugshot Dataset](https://mlsanity.com/paper/2404.11236/) (2024)
- [CemiFace: Center-based Semi-hard Synthetic Face Generation for Face Recognition](https://mlsanity.com/paper/2409.18876/) (2024)
- [Digi2Real: Bridging the Realism Gap in Synthetic Data Face Recognition via Foundation Models](https://mlsanity.com/paper/2411.02188/) (2024)
- [VariFace: Fair and Diverse Synthetic Dataset Generation for Face Recognition](https://mlsanity.com/paper/2412.06235/) (2024)
- [Beyond Real Faces: Synthetic Datasets Can Achieve Reliable Recognition Performance without Privacy Compromise](https://mlsanity.com/paper/2510.17372/) (2025)
- [Privacy-Aware Detection of Fake Identity Documents: Methodology, Benchmark, and Improved Algorithms (FakeIDet2)](https://mlsanity.com/paper/2508.11716/) (2025)
- [SDFD: Building a Versatile Synthetic Face Image Dataset with Diverse Attributes](https://mlsanity.com/paper/2404.17255/) (2024)
- [Personalized Weight Loss Management through Wearable Devices and Artificial Intelligence](https://mlsanity.com/paper/2409.08700/) (2024)
- [VIGFace: Virtual Identity Generation for Privacy-Free Face Recognition](https://mlsanity.com/paper/2403.08277/) (2024)
- [Synthesizing Iris Images using Generative Adversarial Networks: Survey and Comparative Analysis](https://mlsanity.com/paper/2404.17105/) (2024)
- [Synthetic to Authentic: Transferring Realism to 3D Face Renderings for Boosting Face Recognition](https://mlsanity.com/paper/2407.07627/) (2024)
- [From Pixels to Words: Leveraging Explainability in Face Recognition through Interactive Natural Language Processing](https://mlsanity.com/paper/2409.16089/) (2024)
- [UniF2ace: A Unified Fine-grained Face Understanding and Generation Model](https://mlsanity.com/paper/2503.08120/) (2025)
- [If It's Not Enough, Make It So: Reducing Authentic Data Demand in Face Recognition through Synthetic Faces](https://mlsanity.com/paper/2404.03537/) (2024)
- [The Impact of Balancing Real and Synthetic Data on Accuracy and Fairness in Face Recognition](https://mlsanity.com/paper/2409.02867/) (2024)
- [SynMorph: Generating Synthetic Face Morphing Dataset with Mated Samples](https://mlsanity.com/paper/2409.05595/) (2024)
- [Unveiling Synthetic Faces: How Synthetic Datasets Can Expose Real Identities](https://mlsanity.com/paper/2410.24015/) (2024)
- [MINT-Demo: Membership Inference Test Demonstrator](https://mlsanity.com/paper/2503.08332/) (2025)
- [AugGen: Synthetic Augmentation using Diffusion Models Can Improve Recognition](https://mlsanity.com/paper/2503.11544/) (2025)

## Abstract (arXiv)

Face recognition systems have significantly advanced in recent years, driven by the availability of large-scale datasets. However, several issues have recently came up, including privacy concerns that have led to the discontinuation of well-established public datasets. Synthetic datasets have emerged as a solution, even though current synthesis methods present other drawbacks such as limited intra-class variations, lack of realism, and unfair representation of demographic groups. This study introduces GANDiffFace, a novel framework for the generation of synthetic datasets for face recognition that combines the power of Generative Adversarial Networks (GANs) and Diffusion models to overcome the limitations of existing synthetic datasets. In GANDiffFace, we first propose the use of GANs to synthesize highly realistic identities and meet target demographic distributions. Subsequently, we fine-tune Diffusion models with the images generated with GANs, synthesizing multiple images of the same identity with a variety of accessories, poses, expressions, and contexts. We generate multiple synthetic datasets by changing GANDiffFace settings, and compare their mated and non-mated score distributions with the distributions provided by popular real-world datasets for face recognition, i.e. VGG2 and IJB-C. Our results show the feasibility of the proposed GANDiffFace, in particular the use of Diffusion models to enhance the (limited) intra-class variations provided by GANs towards the level of real-world datasets.