# Label-Only Model Inversion Attacks via Knowledge Transfer

arXiv: https://arxiv.org/abs/2310.19342 · HTML: https://mlsanity.com/paper/2310.19342/

## Key facts

- Cited by 21 papers in the map (data through 14 Oct 2026).
- Builds on 41 of its 54 references that are in the map.
- First posted on arXiv 30 Oct 2023.

## Builds on (most cited first, 30 of 41)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Very Deep Convolutional Networks for Large-Scale Image Recognition](https://mlsanity.com/paper/1409.1556/) (2014)
- [Densely Connected Convolutional Networks](https://mlsanity.com/paper/1608.06993/) (2016)
- [EfficientNet: Rethinking Model Scaling for Convolutional Neural Networks](https://mlsanity.com/paper/1905.11946/) (2019)
- [A Style-Based Generator Architecture for Generative Adversarial Networks](https://mlsanity.com/paper/1812.04948/) (2018)
- [Generative Adversarial Networks](https://mlsanity.com/paper/2203.00667/) (2022)
- [SGDR: Stochastic Gradient Descent with Warm Restarts](https://mlsanity.com/paper/1608.03983/) (2016)
- [MobileNetV2: Inverted Residuals and Linear Bottlenecks](https://mlsanity.com/paper/1801.04381/) (2018)
- [Deep Learning Face Attributes in the Wild](https://mlsanity.com/paper/1411.7766/) (2014)
- [Analyzing and Improving the Image Quality of StyleGAN](https://mlsanity.com/paper/1912.04958/) (2019)
- [FaceNet: A Unified Embedding for Face Recognition and Clustering](https://mlsanity.com/paper/1503.03832/) (2015)
- [Supervised Contrastive Learning](https://mlsanity.com/paper/2004.11362/) (2020)
- [Spectral Normalization for Generative Adversarial Networks](https://mlsanity.com/paper/1802.05957/) (2018)
- [A Closer Look at Memorization in Deep Networks](https://mlsanity.com/paper/1706.05394/) (2017)
- [Conditional Image Synthesis With Auxiliary Classifier GANs](https://mlsanity.com/paper/1610.09585/) (2016)
- [MagFace: A Universal Representation for Face Recognition and Quality Assessment](https://mlsanity.com/paper/2103.06627/) (2021)
- [CurricularFace: Adaptive Curriculum Learning Loss for Deep Face Recognition](https://mlsanity.com/paper/2004.00288/) (2020)
- [cGANs with Projection Discriminator](https://mlsanity.com/paper/1802.05637/) (2018)
- [The Secret Revealer: Generative Model-Inversion Attacks Against Deep Neural Networks](https://mlsanity.com/paper/1911.07135/) (2019)
- [Variational Model Inversion Attacks](https://mlsanity.com/paper/2201.10787/) (2022)
- [A Closer Look at Fourier Spectrum Discrepancies for CNN-generated Images Detection](https://mlsanity.com/paper/2103.17195/) (2021)
- [Knowledge-Enriched Distributional Model Inversion Attacks](https://mlsanity.com/paper/2010.04092/) (2020)
- [Plug & Play Attacks: Towards Robust and Flexible Model Inversion Attacks](https://mlsanity.com/paper/2201.12179/) (2022)
- [Towards Data-Free Model Stealing in a Hard Label Setting](https://mlsanity.com/paper/2204.11022/) (2022)
- [Label-Only Model Inversion Attacks via Boundary Repulsion](https://mlsanity.com/paper/2203.01925/) (2022)
- [Improving Robustness to Model Inversion Attacks via Mutual Information Regularization](https://mlsanity.com/paper/2009.05241/) (2020)
- [Contrastive Learning with Continuous Proxy Meta-Data for 3D MRI Classification](https://mlsanity.com/paper/2106.08808/) (2021)
- [Re-thinking Model Inversion Attacks Against Deep Neural Networks](https://mlsanity.com/paper/2304.01669/) (2023)
- [Few-shot Image Generation via Adaptation-Aware Kernel Modulation](https://mlsanity.com/paper/2210.16559/) (2022)
- [Reinforcement Learning-Based Black-Box Model Inversion Attacks](https://mlsanity.com/paper/2304.04625/) (2023)

## Built on by (most cited first, 21 of 21)

- [Privacy Leakage on DNNs: A Survey of Model Inversion Attacks and Defenses](https://mlsanity.com/paper/2402.04013/) (2024)
- [Model Inversion Robustness: Can Transfer Learning Help?](https://mlsanity.com/paper/2405.05588/) (2024)
- [Model Inversion Attacks: A Survey of Approaches and Countermeasures](https://mlsanity.com/paper/2411.10023/) (2024)
- [Model Inversion Attacks Through Target-Specific Conditional Diffusion Models](https://mlsanity.com/paper/2407.11424/) (2024)
- [DORY: Deliberative Prompt Recovery for LLM](https://mlsanity.com/paper/2405.20657/) (2024)
- [Theoretical Insights in Model Inversion Robustness and Conditional Entropy Maximization for Collaborative Inference Systems](https://mlsanity.com/paper/2503.00383/) (2025)
- [MIBench: A Comprehensive Framework for Benchmarking Model Inversion Attack and Defense](https://mlsanity.com/paper/2410.05159/) (2024)
- [Rank Matters: Understanding and Defending Model Inversion Attacks via Low-Rank Feature Filtering](https://mlsanity.com/paper/2410.05814/) (2024)
- [DiffMI: Breaking Face Recognition Privacy via Diffusion-Driven Training-Free Model Inversion](https://mlsanity.com/paper/2504.18015/) (2025)
- [Prediction Exposes Your Face: Black-box Model Inversion via Prediction Alignment](https://mlsanity.com/paper/2407.08127/) (2024)
- [A Privacy-Preserving Federated Learning Method with Homomorphic Encryption in Omics Data](https://mlsanity.com/paper/2511.06064/) (2025)
- [Do Spikes Protect Privacy? Investigating Black-Box Model Inversion Attacks in Spiking Neural Networks](https://mlsanity.com/paper/2502.05509/) (2025)
- [Revisiting Model Inversion Evaluation: From Misleading Standards to Reliable Privacy Assessment](https://mlsanity.com/paper/2505.03519/) (2025)
- [Inverting Black-Box Face Recognition Systems via Zero-Order Optimization in Eigenface Space](https://mlsanity.com/paper/2506.09777/) (2025)
- [ToFU: Transforming How Federated Learning Systems Forget User Data](https://mlsanity.com/paper/2509.15861/) (2025)
- [Generative Model Inversion Through the Lens of the Manifold Hypothesis](https://mlsanity.com/paper/2509.20177/) (2025)
- [AI Security in the Foundation Model Era: A Comprehensive Survey from a Unified Perspective](https://mlsanity.com/paper/2603.24857/) (2026)
- [Do Vision-Language Models Leak What They Learn? Adaptive Token-Weighted Model Inversion Attacks](https://mlsanity.com/paper/2508.04097/) (2025)
- [Model Inversion Attack Against Deep Hashing](https://mlsanity.com/paper/2511.12233/) (2025)
- [Reducing information dependency does not cause training data privacy. Adversarially non-robust features do](https://mlsanity.com/paper/2607.12354/) (2026)
- [On the Relationship between Model Quantization and Model Inversion Attacks](https://mlsanity.com/paper/2610.00382/) (2026)

## Abstract (arXiv)

In a model inversion (MI) attack, an adversary abuses access to a machine learning (ML) model to infer and reconstruct private training data. Remarkable progress has been made in the white-box and black-box setups, where the adversary has access to the complete model or the model's soft output respectively. However, there is very limited study in the most challenging but practically important setup: Label-only MI attacks, where the adversary only has access to the model's predicted label (hard label) without confidence scores nor any other model information. In this work, we propose LOKT, a novel approach for label-only MI attacks. Our idea is based on transfer of knowledge from the opaque target model to surrogate models. Subsequently, using these surrogate models, our approach can harness advanced white-box attacks. We propose knowledge transfer based on generative modelling, and introduce a new model, Target model-assisted ACGAN (T-ACGAN), for effective knowledge transfer. Our method casts the challenging label-only MI into the more tractable white-box setup. We provide analysis to support that surrogate models based on our approach serve as effective proxies for the target model for MI. Our experiments show that our method significantly outperforms existing SOTA Label-only MI attack by more than 15% across all MI benchmarks. Furthermore, our method compares favorably in terms of query budget. Our study highlights rising privacy threats for ML models even when minimal information (i.e., hard labels) is exposed. Our study highlights rising privacy threats for ML models even when minimal information (i.e., hard labels) is exposed. Our code, demo, models and reconstructed data are available at our project page: https://ngoc-nguyen-0.github.io/lokt/