# Controllable Inversion of Black-Box Face Recognition Models via Diffusion

arXiv: https://arxiv.org/abs/2303.13006 · HTML: https://mlsanity.com/paper/2303.13006/

## Key facts

- Cited by 22 papers in the map (data through 14 Oct 2026).
- Placed in Image generation, text-to-image and editing, one of 14 directions in Diffusion models.
- Builds on 55 of its 85 references that are in the map.
- First posted on arXiv 23 Mar 2023.

## Placed in

- Diffusion models / [Image generation, text-to-image and editing](https://mlsanity.com/diffusion/image-generation-text-to-image-and-editing/)

## Builds on (most cited first, 30 of 55)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [Denoising Diffusion Probabilistic Models](https://mlsanity.com/paper/2006.11239/) (2020)
- [U-Net: Convolutional Networks for Biomedical Image Segmentation](https://mlsanity.com/paper/1505.04597/) (2015)
- [High-Resolution Image Synthesis with Latent Diffusion Models](https://mlsanity.com/paper/2112.10752/) (2021)
- [The Unreasonable Effectiveness of Deep Features as a Perceptual Metric](https://mlsanity.com/paper/1801.03924/) (2018)
- [Score-Based Generative Modeling through Stochastic Differential Equations](https://mlsanity.com/paper/2011.13456/) (2020)
- [Diffusion Models Beat GANs on Image Synthesis](https://mlsanity.com/paper/2105.05233/) (2021)
- [Explaining and Harnessing Adversarial Examples](https://mlsanity.com/paper/1412.6572/) (2014)
- [Hierarchical Text-Conditional Image Generation with CLIP Latents](https://mlsanity.com/paper/2204.06125/) (2022)
- [Photorealistic Text-to-Image Diffusion Models with Deep Language Understanding](https://mlsanity.com/paper/2205.11487/) (2022)
- [A Style-Based Generator Architecture for Generative Adversarial Networks](https://mlsanity.com/paper/1812.04948/) (2018)
- [Classifier-Free Diffusion Guidance](https://mlsanity.com/paper/2207.12598/) (2022)
- [Improved Denoising Diffusion Probabilistic Models](https://mlsanity.com/paper/2102.09672/) (2021)
- [Analyzing and Improving the Image Quality of StyleGAN](https://mlsanity.com/paper/1912.04958/) (2019)
- [Elucidating the Design Space of Diffusion-Based Generative Models](https://mlsanity.com/paper/2206.00364/) (2022)
- [FaceNet: A Unified Embedding for Face Recognition and Clustering](https://mlsanity.com/paper/1503.03832/) (2015)
- [Deep Inside Convolutional Networks: Visualising Image Classification Models and Saliency Maps](https://mlsanity.com/paper/1312.6034/) (2013)
- [Image Super-Resolution via Iterative Refinement](https://mlsanity.com/paper/2104.07636/) (2021)
- [Diffusion Posterior Sampling for General Noisy Inverse Problems](https://mlsanity.com/paper/2209.14687/) (2022)
- [Palette: Image-to-Image Diffusion Models](https://mlsanity.com/paper/2111.05826/) (2021)
- [StarGAN v2: Diverse Image Synthesis for Multiple Domains](https://mlsanity.com/paper/1912.01865/) (2019)
- [CosFace: Large Margin Cosine Loss for Deep Face Recognition](https://mlsanity.com/paper/1801.09414/) (2018)
- [Denoising Diffusion Restoration Models](https://mlsanity.com/paper/2201.11793/) (2022)
- [Improved Precision and Recall Metric for Assessing Generative Models](https://mlsanity.com/paper/1904.06991/) (2019)
- [Joint Face Detection and Alignment using Multi-task Cascaded Convolutional Networks](https://mlsanity.com/paper/1604.02878/) (2016)
- [SphereFace: Deep Hypersphere Embedding for Face Recognition](https://mlsanity.com/paper/1704.08063/) (2017)
- [ILVR: Conditioning Method for Denoising Diffusion Probabilistic Models](https://mlsanity.com/paper/2108.02938/) (2021)
- [Improving Diffusion Models for Inverse Problems using Manifold Constraints](https://mlsanity.com/paper/2206.00941/) (2022)

## Built on by (most cited first, 22 of 22)

- [CADS: Unleashing the Diversity of Diffusion Models through Condition-Annealed Sampling](https://mlsanity.com/paper/2310.17347/) (2023)
- [Arc2Face: A Foundation Model for ID-Consistent Human Faces](https://mlsanity.com/paper/2403.11641/) (2024)
- [DiffProtect: Generate Adversarial Examples with Diffusion Models for Facial Privacy Protection](https://mlsanity.com/paper/2305.13625/) (2023)
- [FRCSyn Challenge at WACV 2024:Face Recognition Challenge in the Era of Synthetic Data](https://mlsanity.com/paper/2311.10476/) (2023)
- [Second FRCSyn-onGoing: Winning Solutions and Post-Challenge Analysis to Improve Face Recognition with Synthetic Data](https://mlsanity.com/paper/2412.01383/) (2024)
- [ONOT: a High-Quality ICAO-compliant Synthetic Mugshot Dataset](https://mlsanity.com/paper/2404.11236/) (2024)
- [CemiFace: Center-based Semi-hard Synthetic Face Generation for Face Recognition](https://mlsanity.com/paper/2409.18876/) (2024)
- [SlerpFace: Face Template Protection via Spherical Linear Interpolation](https://mlsanity.com/paper/2407.03043/) (2024)
- [VariFace: Fair and Diverse Synthetic Dataset Generation for Face Recognition](https://mlsanity.com/paper/2412.06235/) (2024)
- [LogicNet: A Logical Consistency Embedded Face Attribute Learning Network](https://mlsanity.com/paper/2311.11208/) (2023)
- [A Comprehensive Survey on Diffusion Models and Their Applications](https://mlsanity.com/paper/2408.10207/) (2024)
- [A Sample-Level Evaluation and Generative Framework for Model Inversion Attacks](https://mlsanity.com/paper/2502.19070/) (2025)
- [DiffMI: Breaking Face Recognition Privacy via Diffusion-Driven Training-Free Model Inversion](https://mlsanity.com/paper/2504.18015/) (2025)
- [IDFace: Face Template Protection for Efficient and Secure Identification](https://mlsanity.com/paper/2507.12050/) (2025)
- [Non-Adaptive Adversarial Face Generation](https://mlsanity.com/paper/2507.12107/) (2025)
- [2D-3D Attention and Entropy for Pose Robust 2D Facial Recognition](https://mlsanity.com/paper/2505.09073/) (2025)
- [Inverting Black-Box Face Recognition Systems via Zero-Order Optimization in Eigenface Space](https://mlsanity.com/paper/2506.09777/) (2025)
- [Model Inversion meets Cryptographic Fuzzy Extractors](https://mlsanity.com/paper/2510.25687/) (2025)
- [StyleYourSmile: Cross-Domain Face Retargeting Without Paired Multi-Style Data](https://mlsanity.com/paper/2512.01895/) (2025)
- [From Measurement to Mitigation: Quantifying and Reducing Identity Leakage in Image Representation Encoders with Linear Subspace Removal](https://mlsanity.com/paper/2604.05296/) (2026)
- [EventFace: Event-Based Face Recognition via Structure-Driven Spatiotemporal Modeling](https://mlsanity.com/paper/2604.06782/) (2026)
- [Unmasking Face Embeddings: Reading, Rendering and Naming with Foundation Models](https://mlsanity.com/paper/2609.00411/) (2026)

## Abstract (arXiv)

Face recognition models embed a face image into a low-dimensional identity vector containing abstract encodings of identity-specific facial features that allow individuals to be distinguished from one another. We tackle the challenging task of inverting the latent space of pre-trained face recognition models without full model access (i.e. black-box setting). A variety of methods have been proposed in literature for this task, but they have serious shortcomings such as a lack of realistic outputs and strong requirements for the data set and accessibility of the face recognition model. By analyzing the black-box inversion problem, we show that the conditional diffusion model loss naturally emerges and that we can effectively sample from the inverse distribution even without an identity-specific loss. Our method, named identity denoising diffusion probabilistic model (ID3PM), leverages the stochastic nature of the denoising diffusion process to produce high-quality, identity-preserving face images with various backgrounds, lighting, poses, and expressions. We demonstrate state-of-the-art performance in terms of identity preservation and diversity both qualitatively and quantitatively, and our method is the first black-box face recognition model inversion method that offers intuitive control over the generation process.