# DiffMI: Breaking Face Recognition Privacy via Diffusion-Driven Training-Free Model Inversion

arXiv: https://arxiv.org/abs/2504.18015 · HTML: https://mlsanity.com/paper/2504.18015/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Builds on 28 of its 67 references that are in the map.
- First posted on arXiv 25 Apr 2025.

## Builds on (most cited first, 28 of 28)

- [High-Resolution Image Synthesis with Latent Diffusion Models](https://mlsanity.com/paper/2112.10752/) (2021)
- [The Unreasonable Effectiveness of Deep Features as a Perceptual Metric](https://mlsanity.com/paper/1801.03924/) (2018)
- [A Style-Based Generator Architecture for Generative Adversarial Networks](https://mlsanity.com/paper/1812.04948/) (2018)
- [Analyzing and Improving the Image Quality of StyleGAN](https://mlsanity.com/paper/1912.04958/) (2019)
- [FaceNet: A Unified Embedding for Face Recognition and Clustering](https://mlsanity.com/paper/1503.03832/) (2015)
- [SDEdit: Guided Image Synthesis and Editing with Stochastic Differential Equations](https://mlsanity.com/paper/2108.01073/) (2021)
- [Reliable evaluation of adversarial robustness with an ensemble of diverse parameter-free attacks](https://mlsanity.com/paper/2003.01690/) (2020)
- [VGGFace2: A dataset for recognising faces across pose and age](https://mlsanity.com/paper/1710.08092/) (2017)
- [Joint Face Detection and Alignment using Multi-task Cascaded Convolutional Networks](https://mlsanity.com/paper/1604.02878/) (2016)
- [Square Attack: a query-efficient black-box adversarial attack via random search](https://mlsanity.com/paper/1912.00049/) (2019)
- [MaskGAN: Towards Diverse and Interactive Facial Image Manipulation](https://mlsanity.com/paper/1907.11922/) (2019)
- [MS-Celeb-1M: A Dataset and Benchmark for Large-Scale Face Recognition](https://mlsanity.com/paper/1607.08221/) (2016)
- [The Secret Revealer: Generative Model-Inversion Attacks Against Deep Neural Networks](https://mlsanity.com/paper/1911.07135/) (2019)
- [On the Reconstruction of Face Images from Deep Face Templates](https://mlsanity.com/paper/1703.00832/) (2017)
- [Plug & Play Attacks: Towards Robust and Flexible Model Inversion Attacks](https://mlsanity.com/paper/2201.12179/) (2022)
- [Vec2Face: Unveil Human Faces from their Blackbox Features in Face Recognition](https://mlsanity.com/paper/2003.06958/) (2020)
- [Privacy-Preserving Face Recognition with Learnable Privacy Budgets in Frequency Domain](https://mlsanity.com/paper/2207.07316/) (2022)
- [Pseudo Label-Guided Model Inversion Attack via Conditional Generative Adversarial Network](https://mlsanity.com/paper/2302.09814/) (2023)
- [Controllable Inversion of Black-Box Face Recognition Models via Diffusion](https://mlsanity.com/paper/2303.13006/) (2023)
- [Label-Only Model Inversion Attacks via Knowledge Transfer](https://mlsanity.com/paper/2310.19342/) (2023)
- [A Closer Look at GAN Priors: Exploiting Intermediate Features for Enhanced Model Inversion Attacks](https://mlsanity.com/paper/2407.13863/) (2024)
- [Privacy-Preserving Face Recognition Using Random Frequency Components](https://mlsanity.com/paper/2308.10461/) (2023)
- [Unstoppable Attack: Label-Only Model Inversion via Conditional Diffusion Model](https://mlsanity.com/paper/2307.08424/) (2023)
- [Similarity-based Gray-box Adversarial Attack Against Deep Face Recognition](https://mlsanity.com/paper/2201.04011/) (2022)
- [Model Inversion Attacks Through Target-Specific Conditional Diffusion Models](https://mlsanity.com/paper/2407.11424/) (2024)
- [A Multi-task Adversarial Attack Against Face Authentication](https://mlsanity.com/paper/2408.08205/) (2024)
- [GreedyPixel: Fine-Grained Black-Box Adversarial Attack Via Greedy Algorithm](https://mlsanity.com/paper/2501.14230/) (2025)
- [Prediction Exposes Your Face: Black-box Model Inversion via Prediction Alignment](https://mlsanity.com/paper/2407.08127/) (2024)

## Built on by (most cited first, 3 of 3)

- [Inverting Black-Box Face Recognition Systems via Zero-Order Optimization in Eigenface Space](https://mlsanity.com/paper/2506.09777/) (2025)
- [Semantic Leakage from Image Embeddings](https://mlsanity.com/paper/2601.22929/) (2026)
- [From Measurement to Mitigation: Quantifying and Reducing Identity Leakage in Image Representation Encoders with Linear Subspace Removal](https://mlsanity.com/paper/2604.05296/) (2026)

## Abstract (arXiv)

Face recognition poses serious privacy risks due to its reliance on sensitive and immutable biometric data. While modern systems mitigate privacy risks by mapping facial images to embeddings (commonly regarded as privacy-preserving), model inversion attacks reveal that identity information can still be recovered, exposing critical vulnerabilities. However, existing attacks are often computationally expensive and lack generalization, especially those requiring target-specific training. Even training-free approaches suffer from limited identity controllability, hindering faithful reconstruction of nuanced or unseen identities. In this work, we propose DiffMI, the first diffusion-driven, training-free model inversion attack. DiffMI introduces a novel pipeline combining robust latent code initialization, a ranked adversarial refinement strategy, and a statistically grounded, confidence-aware optimization objective. DiffMI applies directly to unseen target identities and face recognition models, offering greater adaptability than training-dependent approaches while significantly reducing computational overhead. Our method achieves 84.42%--92.87% attack success rates against inversion-resilient systems and outperforms the best prior training-free GAN-based approach by 4.01%--9.82%. The implementation is available at https://github.com/azrealwang/DiffMI.