# Unstoppable Attack: Label-Only Model Inversion via Conditional Diffusion Model

arXiv: https://arxiv.org/abs/2307.08424 · HTML: https://mlsanity.com/paper/2307.08424/

## Key facts

- Cited by 14 papers in the map (data through 14 Oct 2026).
- Placed in Robustness, privacy and adversarial aspects, one of 14 directions in Diffusion models.
- Builds on 26 of its 53 references that are in the map.
- First posted on arXiv 17 Jul 2023.

## Placed in

- Diffusion models / [Robustness, privacy and adversarial aspects](https://mlsanity.com/diffusion/robustness-privacy-and-adversarial-aspects/)

## Builds on (most cited first, 26 of 26)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Denoising Diffusion Probabilistic Models](https://mlsanity.com/paper/2006.11239/) (2020)
- [Very Deep Convolutional Networks for Large-Scale Image Recognition](https://mlsanity.com/paper/1409.1556/) (2014)
- [The Unreasonable Effectiveness of Deep Features as a Perceptual Metric](https://mlsanity.com/paper/1801.03924/) (2018)
- [Diffusion Models Beat GANs on Image Synthesis](https://mlsanity.com/paper/2105.05233/) (2021)
- [Deep Unsupervised Learning using Nonequilibrium Thermodynamics](https://mlsanity.com/paper/1503.03585/) (2015)
- [Classifier-Free Diffusion Guidance](https://mlsanity.com/paper/2207.12598/) (2022)
- [Generative Adversarial Networks](https://mlsanity.com/paper/2203.00667/) (2022)
- [Deep Learning Face Attributes in the Wild](https://mlsanity.com/paper/1411.7766/) (2014)
- [Membership Inference Attacks against Machine Learning Models](https://mlsanity.com/paper/1610.05820/) (2016)
- [Conditional Generative Adversarial Nets](https://mlsanity.com/paper/1411.1784/) (2014)
- [Stealing Machine Learning Models via Prediction APIs](https://mlsanity.com/paper/1609.02943/) (2016)
- [The Secret Revealer: Generative Model-Inversion Attacks Against Deep Neural Networks](https://mlsanity.com/paper/1911.07135/) (2019)
- [Machine Learning Models that Remember Too Much](https://mlsanity.com/paper/1709.07886/) (2017)
- [Seeing What a GAN Cannot Generate](https://mlsanity.com/paper/1910.11626/) (2019)
- [Knowledge-Enriched Distributional Model Inversion Attacks](https://mlsanity.com/paper/2010.04092/) (2020)
- [Plug & Play Attacks: Towards Robust and Flexible Model Inversion Attacks](https://mlsanity.com/paper/2201.12179/) (2022)
- [Label-Only Model Inversion Attacks via Boundary Repulsion](https://mlsanity.com/paper/2203.01925/) (2022)
- [Improving Robustness to Model Inversion Attacks via Mutual Information Regularization](https://mlsanity.com/paper/2009.05241/) (2020)
- [Reinforcement Learning-Based Black-Box Model Inversion Attacks](https://mlsanity.com/paper/2304.04625/) (2023)
- [Bilateral Dependency Optimization: Defending Against Model-inversion Attacks](https://mlsanity.com/paper/2206.05483/) (2022)
- [Pseudo Label-Guided Model Inversion Attack via Conditional Generative Adversarial Network](https://mlsanity.com/paper/2302.09814/) (2023)
- [Purifier: Defending Data Inference Attacks via Transforming Confidence Scores](https://mlsanity.com/paper/2212.00612/) (2022)
- [Model Inversion Attack against Transfer Learning: Inverting a Model without Accessing It](https://mlsanity.com/paper/2203.06570/) (2022)
- [Targeted Image Reconstruction by Sampling Pre-trained Diffusion Model](https://mlsanity.com/paper/2301.07557/) (2023)

## Built on by (most cited first, 14 of 14)

- [Privacy Leakage on DNNs: A Survey of Model Inversion Attacks and Defenses](https://mlsanity.com/paper/2402.04013/) (2024)
- [Model Inversion Attacks Through Target-Specific Conditional Diffusion Models](https://mlsanity.com/paper/2407.11424/) (2024)
- [SoK: Pitfalls in Evaluating Black-Box Attacks](https://mlsanity.com/paper/2310.17534/) (2023)
- [Trap-MID: Trapdoor-based Defense against Model Inversion Attacks](https://mlsanity.com/paper/2411.08460/) (2024)
- [DiffMI: Breaking Face Recognition Privacy via Diffusion-Driven Training-Free Model Inversion](https://mlsanity.com/paper/2504.18015/) (2025)
- [Minimizing the Pretraining Gap: Domain-aligned Text-Based Person Retrieval](https://mlsanity.com/paper/2507.10195/) (2025)
- [Turning Black Box into White Box: Dataset Distillation Leaks](https://mlsanity.com/paper/2603.01053/) (2026)
- [Rethinking Model Inversion Attacks With Patch-Wise Reconstruction](https://mlsanity.com/paper/2312.07040/) (2023)
- [Machine Unlearning in Forgettability Sequence](https://mlsanity.com/paper/2410.06446/) (2024)
- [Securing Transfer-Learned Networks with Reverse Homomorphic Encryption](https://mlsanity.com/paper/2505.14323/) (2025)
- [Model Inversion Attack Against Deep Hashing](https://mlsanity.com/paper/2511.12233/) (2025)
- [Quality-Preserving Imperceptible Adversarial Attack on Skeleton-based Human Action Recognition](https://mlsanity.com/paper/2606.13022/) (2026)
- [Reducing information dependency does not cause training data privacy. Adversarially non-robust features do](https://mlsanity.com/paper/2607.12354/) (2026)
- [On the Relationship between Model Quantization and Model Inversion Attacks](https://mlsanity.com/paper/2610.00382/) (2026)

## Abstract (arXiv)

Model inversion attacks (MIAs) aim to recover private data from inaccessible training sets of deep learning models, posing a privacy threat. MIAs primarily focus on the white-box scenario where attackers have full access to the model's structure and parameters. However, practical applications are usually in black-box scenarios or label-only scenarios, i.e., the attackers can only obtain the output confidence vectors or labels by accessing the model. Therefore, the attack models in existing MIAs are difficult to effectively train with the knowledge of the target model, resulting in sub-optimal attacks. To the best of our knowledge, we pioneer the research of a powerful and practical attack model in the label-only scenario. In this paper, we develop a novel MIA method, leveraging a conditional diffusion model (CDM) to recover representative samples under the target label from the training set. Two techniques are introduced: selecting an auxiliary dataset relevant to the target model task and using predicted labels as conditions to guide training CDM; and inputting target label, pre-defined guidance strength, and random noise into the trained attack model to generate and correct multiple results for final selection. This method is evaluated using Learned Perceptual Image Patch Similarity as a new metric and as a judgment basis for deciding the values of hyper-parameters. Experimental results show that this method can generate similar and accurate samples to the target label, outperforming generators of previous approaches.